2024-02-19T18:36:22.182 INFO:root:teuthology version: 0.0.1.dev260+gb1dac55 2024-02-19T18:36:22.182 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --name pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617 --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617/orig.config.yaml 2024-02-19T18:36:22.233 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T18:36:22.330 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617 branch: wip-batrick-testing-20240215.160715 description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: pdonnell@redhat.com first_in_suite: false job_id: '7566617' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240215.160715 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: f78a58c0ffd401d1493058a1022c35f011d65275 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f78a58c0ffd401d1493058a1022c35f011d65275 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: i64440 sha1: 1ce7ea8877c709fa862eac608550374bd96e166e owner: scheduled_pdonnell@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 2975 sha1: f78a58c0ffd401d1493058a1022c35f011d65275 sleep_before_teardown: 0 suite: fs:upgrade:mds_upgrade_sequence suite_branch: i64440 suite_path: /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 1ce7ea8877c709fa862eac608550374bd96e166e targets: smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM2yDy7yT4TWsPg0hMNtmQ5bTfrGOnlQy57t5RFdWNXEzxamh4TpvjzKRgdv44arNbjXve1OkYOk5Uh5vwhRJ+8= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsTJiI/5NZz48HT59dZd4kjS1jf7gbBr7c25EYpQKt7xHJOrVyhw+MGQXHSQsGLi4ISYOBIKKLOnbJv4yvOxyc= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - full_sequential_finally: - sleep: duration: 3600 - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: b1dac5519c57c269ea98a22fb7729016a1d0e4d2 timestamp: 2024-02-19_18:28:45 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-02-19T18:36:22.330 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa; will attempt to use it 2024-02-19T18:36:22.330 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks 2024-02-19T18:36:22.331 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-19T18:36:22.332 INFO:teuthology.task.internal:Checking packages... 2024-02-19T18:36:22.353 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f78a58c0ffd401d1493058a1022c35f011d65275' 2024-02-19T18:36:22.353 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:36:22.354 INFO:teuthology.packaging:ref: None 2024-02-19T18:36:22.354 INFO:teuthology.packaging:tag: None 2024-02-19T18:36:22.354 INFO:teuthology.packaging:branch: wip-batrick-testing-20240215.160715 2024-02-19T18:36:22.354 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:36:22.354 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240215.160715 2024-02-19T18:36:22.498 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1450.gf78a58c0 2024-02-19T18:36:22.499 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-19T18:36:22.506 INFO:teuthology.task.internal:no buildpackages task found 2024-02-19T18:36:22.506 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-19T18:36:22.522 INFO:teuthology.task.internal:Saving configuration 2024-02-19T18:36:22.540 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-19T18:36:22.547 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T18:36:22.570 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:48.285546', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM2yDy7yT4TWsPg0hMNtmQ5bTfrGOnlQy57t5RFdWNXEzxamh4TpvjzKRgdv44arNbjXve1OkYOk5Uh5vwhRJ+8='} 2024-02-19T18:36:22.594 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:48.286859', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsTJiI/5NZz48HT59dZd4kjS1jf7gbBr7c25EYpQKt7xHJOrVyhw+MGQXHSQsGLi4ISYOBIKKLOnbJv4yvOxyc='} 2024-02-19T18:36:22.594 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-19T18:36:22.600 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-02-19T18:36:22.600 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-02-19T18:36:22.601 INFO:teuthology.run_tasks:Running task console_log... 2024-02-19T18:36:22.667 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6f0139a0d0>, signals=[15]) 2024-02-19T18:36:22.668 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-19T18:36:22.674 INFO:teuthology.task.internal:Opening connections... 2024-02-19T18:36:22.674 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:36:22.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:36:22.763 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-02-19T18:36:22.764 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:36:22.849 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-19T18:36:22.856 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-02-19T18:36:22.873 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-02-19T18:36:22.873 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Stream" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="9" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="9" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el9" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:LOGO="fedora-logo-icon" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T18:36:22.930 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-19T18:36:22.931 INFO:teuthology.orchestra.run.smithi077.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T18:36:22.931 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-02-19T18:36:22.950 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-02-19T18:36:22.966 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-02-19T18:36:22.967 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-19T18:36:23.022 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-19T18:36:23.023 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-19T18:36:23.023 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-02-19T18:36:23.041 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-19T18:36:23.058 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-19T18:36:23.065 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-19T18:36:23.065 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-02-19T18:36:23.068 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-02-19T18:36:23.083 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-19T18:36:23.089 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-19T18:36:23.089 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-02-19T18:36:23.124 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-02-19T18:36:23.450 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-19T18:36:23.502 INFO:teuthology.run_tasks:Running task kernel... 2024-02-19T18:36:23.518 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-19T18:36:23.518 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-19T18:36:23.518 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T18:36:23.519 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-19T18:36:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:36:23.535 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-02-19T18:36:23.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:36:23.536 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-02-19T18:36:23.592 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-361.el9.x86_64 2024-02-19T18:36:23.592 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.14.0-361.el9.x86_64 2024-02-19T18:36:23.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2024-02-19T18:36:23.595 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-361.el9.x86_64 2024-02-19T18:36:23.596 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-361.el9.x86_64 2024-02-19T18:36:23.596 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-02-19T18:36:25.690 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-19T18:36:25.838 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-19T18:36:27.282 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-02-19T18:36:27.480 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-02-19T18:36:29.056 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-02-19T18:36:31.422 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-02-19T18:36:33.365 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-02-19T18:36:38.007 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 5.3 MB/s | 6.1 MB 00:01 2024-02-19T18:36:39.424 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 15 kB 00:00 2024-02-19T18:36:41.586 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:02 2024-02-19T18:36:46.525 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 9.5 kB/s | 1.7 kB 00:00 2024-02-19T18:36:48.782 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-19T18:36:48.782 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-19T18:36:48.884 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:36:48.900 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:36:48.901 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:36:48.902 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:36:48.902 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-02-19T18:36:48.902 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:36:48.902 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 100 M 2024-02-19T18:36:48.902 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 123 M 2024-02-19T18:36:48.902 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:36:49.284 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 16 MB/s | 6.1 MB 00:00 2024-02-19T18:36:49.843 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-19T18:36:51.034 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): kernel-modules-core-5.14.0-421.el9.x86_6 19 MB/s | 33 MB 00:01 2024-02-19T18:36:51.664 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-02-19T18:36:51.668 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:36:51.668 INFO:teuthology.orchestra.run.smithi104.stdout:Total 36 MB/s | 100 MB 00:02 2024-02-19T18:36:51.668 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:36:52.050 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:36:52.050 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:36:53.124 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:36:53.124 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:36:54.786 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 794 kB/s | 19 MB 00:24 2024-02-19T18:36:55.582 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:36:56.235 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:36:56.367 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:36:59.110 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:36:59.162 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 7.3 MB/s | 6.1 MB 00:00 2024-02-19T18:36:59.292 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:37:00.521 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 15 kB 00:00 2024-02-19T18:37:02.017 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 15 MB/s | 20 MB 00:01 2024-02-19T18:37:04.056 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:37:04.162 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:06.881 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-02-19T18:37:08.913 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:09.122 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-19T18:37:09.122 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-19T18:37:09.210 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:37:09.220 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:37:09.220 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 5.14.0-421.el9 CentOS-BaseOS 6.1 M 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core x86_64 5.14.0-421.el9 CentOS-BaseOS 21 M 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules x86_64 5.14.0-421.el9 CentOS-BaseOS 39 M 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-core x86_64 5.14.0-421.el9 CentOS-BaseOS 33 M 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout:Install 4 Packages 2024-02-19T18:37:09.221 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:37:09.222 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 100 M 2024-02-19T18:37:09.222 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 123 M 2024-02-19T18:37:09.222 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:37:09.641 INFO:teuthology.orchestra.run.smithi077.stdout:(1/4): kernel-5.14.0-421.el9.x86_64.rpm 14 MB/s | 6.1 MB 00:00 2024-02-19T18:37:10.216 INFO:teuthology.orchestra.run.smithi077.stdout:(2/4): kernel-core-5.14.0-421.el9.x86_64.rpm 21 MB/s | 21 MB 00:00 2024-02-19T18:37:10.950 INFO:teuthology.orchestra.run.smithi077.stdout:(3/4): kernel-modules-core-5.14.0-421.el9.x86_6 26 MB/s | 33 MB 00:01 2024-02-19T18:37:11.708 INFO:teuthology.orchestra.run.smithi077.stdout:(4/4): kernel-modules-5.14.0-421.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-02-19T18:37:11.712 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:37:11.712 INFO:teuthology.orchestra.run.smithi077.stdout:Total 40 MB/s | 100 MB 00:02 2024-02-19T18:37:11.712 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:37:12.085 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:37:12.085 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:37:13.101 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:37:13.102 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:37:15.477 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:37:16.150 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-core-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:37:16.276 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:37:19.029 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:37:19.213 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:37:23.953 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:37:24.183 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:28.864 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:38.281 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:38.281 INFO:teuthology.orchestra.run.smithi104.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-19T18:37:38.282 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:37:38.349 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:38.734 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:38.734 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:37:38.734 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:37:38.734 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:37:39.130 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:39.131 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:37:39.131 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:37:39.131 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-19T18:37:39.131 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-19T18:37:39.131 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-19T18:37:39.131 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-19T18:37:39.131 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:37:39.131 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:37:39.642 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:37:39.772 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:37:39.773 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:37:39.773 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-19T18:37:39.773 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T18:37:39.773 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T18:37:39.774 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-02-19T18:37:39.774 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T18:37:39.774 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:37:39.883 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:37:39.883 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:37:39.883 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-02-19T18:37:39.923 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-02-19T18:37:39.923 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-02-19T18:37:39.923 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.06 2024-02-19T18:37:39.923 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 68.el9 2024-02-19T18:37:39.923 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-02-19T18:37:39.923 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Group : Unspecified 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 8264254 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : builder@centos.org 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-02-19T18:37:39.924 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:37:39.925 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T18:37:39.925 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T18:37:39.925 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T18:37:39.925 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-02-19T18:37:39.925 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:37:39.925 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T18:37:39.926 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T18:37:39.926 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 5.14.0-421.el9.x86_64 2024-02-19T18:37:39.926 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T18:37:40.509 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-02-19T18:37:41.872 INFO:teuthology.orchestra.run.smithi104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-19T18:37:41.902 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-02-19T18:37:41.911 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-02-19T18:37:41.928 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.R3igKV723c 2024-02-19T18:37:41.928 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.R3igKV723c 2024-02-19T18:37:41.995 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.R3igKV723c 2024-02-19T18:37:42.136 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.R3igKV723c is 6KB 2024-02-19T18:37:42.156 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.R3igKV723c 2024-02-19T18:37:42.170 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T18:37:42.237 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-19T18:37:42.237 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-19T18:37:42.237 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-19T18:37:42.237 INFO:teuthology.orchestra.run.smithi104.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-19T18:37:42.239 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-19T18:37:42.412 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-02-19T18:37:58.396 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:58.396 INFO:teuthology.orchestra.run.smithi077.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-19T18:37:58.397 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:37:58.466 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:58.925 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:58.925 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-5.14.0-421.el9.x86_64 1/4 2024-02-19T18:37:58.925 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-core-5.14.0-421.el9.x86_64 2/4 2024-02-19T18:37:58.925 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-5.14.0-421.el9.x86_64 3/4 2024-02-19T18:37:59.341 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-core-5.14.0-421.el9.x86_64 4/4 2024-02-19T18:37:59.341 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:37:59.342 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:37:59.342 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-5.14.0-421.el9.x86_64 2024-02-19T18:37:59.342 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core-5.14.0-421.el9.x86_64 2024-02-19T18:37:59.342 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-5.14.0-421.el9.x86_64 2024-02-19T18:37:59.342 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-core-5.14.0-421.el9.x86_64 2024-02-19T18:37:59.342 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:37:59.342 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:37:59.874 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:38:00.004 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:38:00.004 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:38:00.004 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-421.el9.x86_64 2024-02-19T18:38:00.005 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-19T18:38:00.005 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-19T18:38:00.005 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-02-19T18:38:00.005 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2024-02-19T18:38:00.005 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel | sort -rV | head -n 1 2024-02-19T18:38:00.112 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-5.14.0-421.el9.x86_64 2024-02-19T18:38:00.112 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-421.el9.x86_64 2024-02-19T18:38:00.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.06 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 68.el9 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Group : Unspecified 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 8264254 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2024-02-19T18:38:00.153 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : builder@centos.org 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2024-02-19T18:38:00.154 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:38:00.155 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2024-02-19T18:38:00.155 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-19T18:38:00.155 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 5.14.0-421.el9.x86_64 2024-02-19T18:38:00.156 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-19T18:38:00.780 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2024-02-19T18:38:02.184 INFO:teuthology.orchestra.run.smithi077.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-19T18:38:02.219 INFO:teuthology.orchestra.run.smithi077.stderr:done 2024-02-19T18:38:02.221 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-02-19T18:38:02.244 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.fWvkUVwSgS 2024-02-19T18:38:02.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.fWvkUVwSgS 2024-02-19T18:38:02.312 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.fWvkUVwSgS 2024-02-19T18:38:02.449 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.fWvkUVwSgS is 6KB 2024-02-19T18:38:02.497 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.fWvkUVwSgS 2024-02-19T18:38:02.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2024-02-19T18:38:02.585 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-19T18:38:02.585 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-19T18:38:02.585 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-19T18:38:02.585 INFO:teuthology.orchestra.run.smithi077.stdout:15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64.conf 2024-02-19T18:38:02.586 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-421.el9.x86_64 2024-02-19T18:38:02.765 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2024-02-19T18:38:12.424 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-02-19T18:38:12.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T18:38:12.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:38:30.973 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T18:38:32.771 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-02-19T18:38:32.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-19T18:38:32.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:38:39.980 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T18:38:39.981 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:38:43.036 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T18:38:51.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-02-19T18:38:55.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T18:38:55.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:39:00.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-19T18:39:00.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:39:03.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-02-19T18:39:13.468 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T18:39:15.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-19T18:39:15.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:39:28.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T18:39:28.481 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:39:28.839 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-02-19T18:39:29.304 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T18:39:29.304 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-421.el9.x86_64"... 2024-02-19T18:39:29.305 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-02-19T18:39:29.319 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-421.el9.x86_64 2024-02-19T18:39:29.319 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-19T18:39:29.319 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T18:39:29.319 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T18:39:30.320 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-02-19T18:39:30.320 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T18:39:30.432 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-02-19T18:39:30.466 DEBUG:teuthology.parallel:result is None 2024-02-19T18:39:33.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-02-19T18:39:48.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-19T18:39:48.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:39:49.065 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-02-19T18:39:49.536 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-19T18:39:49.536 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-421.el9.x86_64"... 2024-02-19T18:39:49.536 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-02-19T18:39:49.550 INFO:teuthology.orchestra.run.smithi077.stdout:5.14.0-421.el9.x86_64 2024-02-19T18:39:49.550 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-421.el9.x86_64 vs 5.14.0-421.el9.x86_64 2024-02-19T18:39:49.550 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-19T18:39:49.550 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-19T18:39:50.551 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-02-19T18:39:50.551 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-19T18:39:50.601 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-02-19T18:39:50.639 DEBUG:teuthology.parallel:result is None 2024-02-19T18:39:50.640 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-19T18:39:50.647 INFO:teuthology.task.internal:Creating test directory... 2024-02-19T18:39:50.647 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T18:39:50.650 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-19T18:39:50.668 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-19T18:39:50.675 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-19T18:39:50.685 INFO:teuthology.task.internal:Creating archive directory... 2024-02-19T18:39:50.686 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T18:39:50.704 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-19T18:39:50.753 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-19T18:39:50.761 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-19T18:39:50.761 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T18:39:50.769 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-19T18:39:50.804 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:39:50.817 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:39:50.827 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:39:50.841 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-19T18:39:50.843 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-19T18:39:50.850 INFO:teuthology.task.internal:Configuring sudo... 2024-02-19T18:39:50.851 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T18:39:50.861 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-19T18:39:50.915 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-19T18:39:50.924 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-19T18:39:50.925 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T18:39:50.928 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-19T18:39:50.970 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:39:51.029 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:39:51.071 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:39:51.153 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:39:51.195 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:39:51.195 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T18:39:51.261 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:39:51.308 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T18:39:51.358 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:39:51.442 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T18:39:51.486 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:39:51.486 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-19T18:39:51.555 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-02-19T18:39:51.557 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-02-19T18:39:51.602 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T18:39:51.646 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T18:39:52.112 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-19T18:39:52.120 INFO:teuthology.task.internal:Starting timer... 2024-02-19T18:39:52.120 INFO:teuthology.run_tasks:Running task pcp... 2024-02-19T18:39:52.131 INFO:teuthology.run_tasks:Running task selinux... 2024-02-19T18:39:52.140 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-19T18:39:52.140 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2024-02-19T18:39:52.195 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: 2024-02-19T18:39:52.196 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-02-19T18:39:52.273 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-02-19T18:39:52.274 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-19T18:39:52.275 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2024-02-19T18:39:52.303 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2024-02-19T18:39:52.304 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-02-19T18:39:52.335 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-02-19T18:39:52.336 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2024-02-19T18:39:52.336 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T18:39:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:39:52.373 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-19T18:39:52.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:39:52.405 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-19T18:39:52.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2024-02-19T18:39:52.448 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-02-19T18:39:52.487 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-19T18:39:52.497 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-19T18:39:52.515 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-19T18:39:52.516 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-19T18:45:18.136 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-02-19T18:45:18.138 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-19T18:45:18.139 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:45:18.468 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-02-19T18:45:18.555 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-02-19T18:45:18.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T18:45:18.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T18:45:18.785 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-02-19T18:45:18.875 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T18:45:18.876 INFO:teuthology.run_tasks:Running task clock... 2024-02-19T18:45:18.893 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-19T18:45:18.894 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T18:45:18.894 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T18:45:18.920 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-19T18:45:18.923 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T18:45:18.953 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T18:45:18.953 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T18:45:18.985 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-19T18:45:18.985 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2024-02-19T18:45:18.997 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2024-02-19T18:45:19.004 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-19T18:45:19.009 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T18:45:19.026 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T18:45:19.034 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-02-19T18:45:19.050 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-02-19T18:45:19.064 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-19T18:45:19.078 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-02-19T18:45:19.080 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-19T18:45:19.080 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T18:45:19.081 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-02-19T18:45:19.081 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:45:19.081 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:45:19.081 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:45:19.081 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:45:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-02-19T18:45:19.153 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T18:45:19.153 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-02-19T18:45:19.153 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:45:19.153 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:45:19.154 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:45:19.154 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T18:45:19.154 INFO:teuthology.run_tasks:Running task install... 2024-02-19T18:45:19.195 DEBUG:teuthology.task.install:project ceph 2024-02-19T18:45:19.195 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'}} 2024-02-19T18:45:19.195 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'} 2024-02-19T18:45:19.196 INFO:teuthology.task.install:Using flavor: default 2024-02-19T18:45:19.216 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-19T18:45:19.217 INFO:teuthology.task.install:extra packages: [] 2024-02-19T18:45:19.217 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'tag': None, 'wait_for_package': False} 2024-02-19T18:45:19.217 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:45:19.217 INFO:teuthology.packaging:ref: None 2024-02-19T18:45:19.218 INFO:teuthology.packaging:tag: None 2024-02-19T18:45:19.218 INFO:teuthology.packaging:branch: quincy 2024-02-19T18:45:19.218 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:45:19.218 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-19T18:45:19.246 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275', 'tag': None, 'wait_for_package': False} 2024-02-19T18:45:19.249 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:45:19.249 INFO:teuthology.packaging:ref: None 2024-02-19T18:45:19.249 INFO:teuthology.packaging:tag: None 2024-02-19T18:45:19.249 INFO:teuthology.packaging:branch: quincy 2024-02-19T18:45:19.249 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:45:19.249 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-19T18:45:19.524 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/ 2024-02-19T18:45:19.524 INFO:teuthology.task.install.rpm:Package version is 17.2.7-521.gc80e7168 2024-02-19T18:45:19.583 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/ 2024-02-19T18:45:19.584 INFO:teuthology.task.install.rpm:Package version is 17.2.7-521.gc80e7168 2024-02-19T18:45:19.644 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T18:45:19.645 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:45:19.645 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T18:45:19.677 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T18:45:19.677 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:45:19.677 INFO:teuthology.packaging:ref: None 2024-02-19T18:45:19.677 INFO:teuthology.packaging:tag: None 2024-02-19T18:45:19.678 INFO:teuthology.packaging:branch: quincy 2024-02-19T18:45:19.678 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:45:19.678 DEBUG:teuthology.orchestra.run.smithi104:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T18:45:19.712 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/c80e7168938c48b49e5ce2e002f48bf150b5f4a4/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-19T18:45:19.713 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:45:19.713 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-19T18:45:19.751 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-02-19T18:45:19.751 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:45:19.751 INFO:teuthology.packaging:ref: None 2024-02-19T18:45:19.752 INFO:teuthology.packaging:tag: None 2024-02-19T18:45:19.752 INFO:teuthology.packaging:branch: quincy 2024-02-19T18:45:19.752 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:45:19.752 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-19T18:45:19.755 DEBUG:teuthology.orchestra.run.smithi104:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T18:45:19.821 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-19T18:45:19.840 DEBUG:teuthology.orchestra.run.smithi104:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T18:45:19.904 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-19T18:45:19.925 INFO:teuthology.orchestra.run.smithi104.stdout:check_obsoletes = 1 2024-02-19T18:45:19.925 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum clean all 2024-02-19T18:45:19.972 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2024-02-19T18:45:19.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2024-02-19T18:45:20.184 INFO:teuthology.orchestra.run.smithi104.stdout:74 files removed 2024-02-19T18:45:20.214 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-radosgw 2024-02-19T18:45:20.215 INFO:teuthology.orchestra.run.smithi077.stdout:74 files removed 2024-02-19T18:45:20.244 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw 2024-02-19T18:45:21.316 INFO:teuthology.orchestra.run.smithi104.stdout:ceph packages for x86_64 99 kB/s | 77 kB 00:00 2024-02-19T18:45:21.355 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 98 kB/s | 77 kB 00:00 2024-02-19T18:45:22.023 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 15 kB/s | 9.1 kB 00:00 2024-02-19T18:45:22.033 INFO:teuthology.orchestra.run.smithi104.stdout:ceph noarch packages 14 kB/s | 9.1 kB 00:00 2024-02-19T18:45:22.647 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 3.0 kB/s | 1.8 kB 00:00 2024-02-19T18:45:22.783 INFO:teuthology.orchestra.run.smithi104.stdout:ceph source packages 2.5 kB/s | 1.8 kB 00:00 2024-02-19T18:45:23.194 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-02-19T18:45:23.250 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-02-19T18:45:24.659 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-02-19T18:45:24.705 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-02-19T18:45:25.712 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-02-19T18:45:28.273 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-02-19T18:45:34.174 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.1 MB 00:02 2024-02-19T18:45:35.057 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - BaseOS 812 kB/s | 8.0 MB 00:10 2024-02-19T18:45:35.705 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 15 kB 00:00 2024-02-19T18:45:37.617 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2024-02-19T18:45:42.309 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - AppStream 3.1 MB/s | 19 MB 00:06 2024-02-19T18:45:42.681 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-02-19T18:45:45.260 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:45:45.270 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 14 M 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrading: 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: librados2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.1 M 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base x86_64 2:17.2.7-521.gc80e7168.el9 ceph 5.0 M 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 23 M 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-19T18:45:45.271 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 635 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 457 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 4.9 M 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 45 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 116 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs x86_64 2:17.2.7-521.gc80e7168.el9 ceph 156 k 2024-02-19T18:45:45.272 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados x86_64 2:17.2.7-521.gc80e7168.el9 ceph 320 k 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 293 k 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 99 k 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout:Install 22 Packages 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout:Upgrade 2 Packages 2024-02-19T18:45:45.273 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:45:45.274 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 58 M 2024-02-19T18:45:45.274 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:45:46.392 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-02-19T18:45:47.482 INFO:teuthology.orchestra.run.smithi104.stdout:(1/24): ceph-radosgw-17.2.7-521.gc80e7168.el9.x 7.5 MB/s | 14 MB 00:01 2024-02-19T18:45:47.699 INFO:teuthology.orchestra.run.smithi104.stdout:(2/24): ceph-common-17.2.7-521.gc80e7168.el9.x8 11 MB/s | 23 MB 00:02 2024-02-19T18:45:47.724 INFO:teuthology.orchestra.run.smithi104.stdout:(3/24): ceph-selinux-17.2.7-521.gc80e7168.el9.x 104 kB/s | 25 kB 00:00 2024-02-19T18:45:47.775 INFO:teuthology.orchestra.run.smithi104.stdout:(4/24): libcephfs2-17.2.7-521.gc80e7168.el9.x86 8.3 MB/s | 635 kB 00:00 2024-02-19T18:45:47.795 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 15 kB 00:00 2024-02-19T18:45:47.858 INFO:teuthology.orchestra.run.smithi104.stdout:(5/24): ceph-base-17.2.7-521.gc80e7168.el9.x86_ 2.2 MB/s | 5.0 MB 00:02 2024-02-19T18:45:47.883 INFO:teuthology.orchestra.run.smithi104.stdout:(6/24): libradosstriper1-17.2.7-521.gc80e7168.e 2.8 MB/s | 457 kB 00:00 2024-02-19T18:45:47.950 INFO:teuthology.orchestra.run.smithi104.stdout:(7/24): python3-ceph-argparse-17.2.7-521.gc80e7 491 kB/s | 45 kB 00:00 2024-02-19T18:45:47.975 INFO:teuthology.orchestra.run.smithi104.stdout:(8/24): python3-ceph-common-17.2.7-521.gc80e716 1.2 MB/s | 116 kB 00:00 2024-02-19T18:45:48.078 INFO:teuthology.orchestra.run.smithi104.stdout:(9/24): librgw2-17.2.7-521.gc80e7168.el9.x86_64 16 MB/s | 4.9 MB 00:00 2024-02-19T18:45:48.101 INFO:teuthology.orchestra.run.smithi104.stdout:(10/24): python3-cephfs-17.2.7-521.gc80e7168.el 1.0 MB/s | 156 kB 00:00 2024-02-19T18:45:48.126 INFO:teuthology.orchestra.run.smithi104.stdout:(11/24): python3-rados-17.2.7-521.gc80e7168.el9 2.1 MB/s | 320 kB 00:00 2024-02-19T18:45:48.152 INFO:teuthology.orchestra.run.smithi104.stdout:(12/24): python3-rbd-17.2.7-521.gc80e7168.el9.x 3.8 MB/s | 293 kB 00:00 2024-02-19T18:45:48.194 INFO:teuthology.orchestra.run.smithi104.stdout:(13/24): python3-rgw-17.2.7-521.gc80e7168.el9.x 1.1 MB/s | 99 kB 00:00 2024-02-19T18:45:48.419 INFO:teuthology.orchestra.run.smithi104.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 165 kB/s | 44 kB 00:00 2024-02-19T18:45:48.461 INFO:teuthology.orchestra.run.smithi104.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 476 kB/s | 159 kB 00:00 2024-02-19T18:45:48.553 INFO:teuthology.orchestra.run.smithi104.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 282 kB 00:00 2024-02-19T18:45:48.570 INFO:teuthology.orchestra.run.smithi104.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 380 kB/s | 41 kB 00:00 2024-02-19T18:45:48.644 INFO:teuthology.orchestra.run.smithi104.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.6 MB/s | 661 kB 00:00 2024-02-19T18:45:49.164 INFO:teuthology.orchestra.run.smithi104.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 82 kB/s | 49 kB 00:00 2024-02-19T18:45:49.203 INFO:teuthology.orchestra.run.smithi104.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 121 kB/s | 67 kB 00:00 2024-02-19T18:45:49.387 INFO:teuthology.orchestra.run.smithi104.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 370 kB/s | 309 kB 00:00 2024-02-19T18:45:49.654 INFO:teuthology.orchestra.run.smithi104.stdout:(22/24): librbd1-17.2.7-521.gc80e7168.el9.x86_6 11 MB/s | 2.9 MB 00:00 2024-02-19T18:45:49.837 INFO:teuthology.orchestra.run.smithi104.stdout:(23/24): librados2-17.2.7-521.gc80e7168.el9.x86 4.9 MB/s | 3.1 MB 00:00 2024-02-19T18:45:50.257 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 20 MB 00:02 2024-02-19T18:45:51.329 INFO:teuthology.orchestra.run.smithi104.stdout:(24/24): thrift-0.15.0-2.el9.x86_64.rpm 747 kB/s | 1.6 MB 00:02 2024-02-19T18:45:51.336 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:45:51.336 INFO:teuthology.orchestra.run.smithi104.stdout:Total 9.5 MB/s | 58 MB 00:06 2024-02-19T18:45:51.337 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:45:51.369 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:45:51.369 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:45:51.828 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:45:51.828 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:45:53.090 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:45:53.422 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-19T18:45:53.481 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-19T18:45:53.555 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-19T18:45:53.642 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-19T18:45:53.665 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-19T18:45:53.745 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-19T18:45:53.790 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-19T18:45:53.820 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-19T18:45:54.252 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-19T18:45:54.285 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-19T18:45:54.354 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-19T18:45:54.384 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-19T18:45:54.428 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 10/26 2024-02-19T18:45:54.464 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-19T18:45:54.527 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-19T18:45:54.554 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-19T18:45:54.620 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-19T18:45:54.660 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-19T18:45:54.704 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-19T18:45:54.736 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-19T18:45:55.031 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-19T18:45:55.088 INFO:teuthology.orchestra.run.smithi104.stdout: Upgrading : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-19T18:45:55.181 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-19T18:45:55.221 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-19T18:45:55.253 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 19/26 2024-02-19T18:45:55.285 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 20/26 2024-02-19T18:45:57.791 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-19T18:45:57.798 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:45:57.816 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:45:57.816 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:45:57.816 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:45:57.816 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:45:57.816 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 14 M 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.1 M 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:17.2.7-521.gc80e7168.el9 ceph 5.0 M 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 23 M 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 635 k 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 457 k 2024-02-19T18:45:57.817 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:17.2.7-521.gc80e7168.el9 ceph 4.9 M 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 45 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:17.2.7-521.gc80e7168.el9 ceph 116 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:17.2.7-521.gc80e7168.el9 ceph 156 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:17.2.7-521.gc80e7168.el9 ceph 320 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 293 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:17.2.7-521.gc80e7168.el9 ceph 99 k 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:45:57.818 INFO:teuthology.orchestra.run.smithi077.stdout:Install 22 Packages 2024-02-19T18:45:57.819 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 2 Packages 2024-02-19T18:45:57.819 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:45:57.819 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 58 M 2024-02-19T18:45:57.819 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:45:57.840 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-19T18:45:58.388 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-19T18:45:58.443 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-19T18:45:58.536 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-19T18:45:58.536 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-19T18:45:58.542 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-19T18:45:58.542 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:45:58.561 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-19T18:45:58.647 INFO:teuthology.orchestra.run.smithi077.stdout:(1/24): ceph-base-17.2.7-521.gc80e7168.el9.x86_ 6.7 MB/s | 5.0 MB 00:00 2024-02-19T18:45:58.714 INFO:teuthology.orchestra.run.smithi077.stdout:(2/24): ceph-selinux-17.2.7-521.gc80e7168.el9.x 379 kB/s | 25 kB 00:00 2024-02-19T18:45:58.806 INFO:teuthology.orchestra.run.smithi077.stdout:(3/24): libcephfs2-17.2.7-521.gc80e7168.el9.x86 6.8 MB/s | 635 kB 00:00 2024-02-19T18:45:58.889 INFO:teuthology.orchestra.run.smithi077.stdout:(4/24): libradosstriper1-17.2.7-521.gc80e7168.e 5.4 MB/s | 457 kB 00:00 2024-02-19T18:45:59.431 INFO:teuthology.orchestra.run.smithi077.stdout:(5/24): librgw2-17.2.7-521.gc80e7168.el9.x86_64 9.1 MB/s | 4.9 MB 00:00 2024-02-19T18:45:59.498 INFO:teuthology.orchestra.run.smithi077.stdout:(6/24): python3-ceph-argparse-17.2.7-521.gc80e7 678 kB/s | 45 kB 00:00 2024-02-19T18:45:59.574 INFO:teuthology.orchestra.run.smithi077.stdout:(7/24): python3-ceph-common-17.2.7-521.gc80e716 1.5 MB/s | 116 kB 00:00 2024-02-19T18:45:59.649 INFO:teuthology.orchestra.run.smithi077.stdout:(8/24): python3-cephfs-17.2.7-521.gc80e7168.el9 2.0 MB/s | 156 kB 00:00 2024-02-19T18:45:59.741 INFO:teuthology.orchestra.run.smithi077.stdout:(9/24): python3-rados-17.2.7-521.gc80e7168.el9. 3.4 MB/s | 320 kB 00:00 2024-02-19T18:45:59.824 INFO:teuthology.orchestra.run.smithi077.stdout:(10/24): python3-rbd-17.2.7-521.gc80e7168.el9.x 3.4 MB/s | 293 kB 00:00 2024-02-19T18:45:59.899 INFO:teuthology.orchestra.run.smithi077.stdout:(11/24): python3-rgw-17.2.7-521.gc80e7168.el9.x 1.3 MB/s | 99 kB 00:00 2024-02-19T18:46:00.008 INFO:teuthology.orchestra.run.smithi077.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-02-19T18:46:00.033 INFO:teuthology.orchestra.run.smithi077.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-02-19T18:46:00.100 INFO:teuthology.orchestra.run.smithi077.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-02-19T18:46:00.142 INFO:teuthology.orchestra.run.smithi077.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-02-19T18:46:00.167 INFO:teuthology.orchestra.run.smithi077.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-02-19T18:46:00.726 INFO:teuthology.orchestra.run.smithi077.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 553 kB/s | 309 kB 00:00 2024-02-19T18:46:00.827 INFO:teuthology.orchestra.run.smithi077.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-02-19T18:46:00.935 INFO:teuthology.orchestra.run.smithi077.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2024-02-19T18:46:01.244 INFO:teuthology.orchestra.run.smithi077.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 5.1 MB/s | 1.6 MB 00:00 2024-02-19T18:46:01.911 INFO:teuthology.orchestra.run.smithi077.stdout:(21/24): ceph-common-17.2.7-521.gc80e7168.el9.x 5.6 MB/s | 23 MB 00:04 2024-02-19T18:46:02.053 INFO:teuthology.orchestra.run.smithi077.stdout:(22/24): ceph-radosgw-17.2.7-521.gc80e7168.el9. 3.4 MB/s | 14 MB 00:04 2024-02-19T18:46:02.111 INFO:teuthology.orchestra.run.smithi077.stdout:(23/24): librados2-17.2.7-521.gc80e7168.el9.x86 3.5 MB/s | 3.1 MB 00:00 2024-02-19T18:46:02.272 INFO:teuthology.orchestra.run.smithi077.stdout:(24/24): librbd1-17.2.7-521.gc80e7168.el9.x86_6 8.2 MB/s | 2.9 MB 00:00 2024-02-19T18:46:02.272 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:46:02.273 INFO:teuthology.orchestra.run.smithi077.stdout:Total 13 MB/s | 58 MB 00:04 2024-02-19T18:46:02.273 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:46:02.303 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:46:02.303 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:46:02.770 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:46:02.771 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:46:04.003 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:46:04.349 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-02-19T18:46:04.405 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-19T18:46:04.488 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-19T18:46:04.568 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-19T18:46:04.598 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-19T18:46:04.672 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-19T18:46:04.714 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-02-19T18:46:04.752 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-02-19T18:46:05.185 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-02-19T18:46:05.209 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-19T18:46:05.259 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 8/26 2024-02-19T18:46:05.284 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-02-19T18:46:05.318 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 10/26 2024-02-19T18:46:05.355 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-19T18:46:05.418 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-19T18:46:05.446 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-19T18:46:05.511 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-19T18:46:05.550 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-02-19T18:46:05.594 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-02-19T18:46:05.627 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-02-19T18:46:05.924 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-02-19T18:46:05.949 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-19T18:46:06.037 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 18/26 2024-02-19T18:46:06.110 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 19/26 2024-02-19T18:46:06.134 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 20/26 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /sys 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /proc 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /mnt 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /var/tmp 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /home 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /root 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout:skipping the directory /tmp 2024-02-19T18:46:07.996 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:08.532 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-19T18:46:08.564 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-19T18:46:09.114 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 21/26 2024-02-19T18:46:09.151 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-19T18:46:09.289 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-19T18:46:09.289 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 22/26 2024-02-19T18:46:09.289 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-19T18:46:09.290 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-19T18:46:09.290 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:09.327 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-19T18:46:09.350 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-19T18:46:09.350 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:09.350 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-19T18:46:09.350 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:46:09.350 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:46:09.350 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:09.351 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-19T18:46:09.442 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-19T18:46:09.442 INFO:teuthology.orchestra.run.smithi104.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 1/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 5/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 6/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 7/26 2024-02-19T18:46:10.661 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 8/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 9/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 10/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-19T18:46:10.662 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-19T18:46:10.663 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-19T18:46:10.663 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-19T18:46:10.663 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 25/26 2024-02-19T18:46:11.663 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-19T18:46:11.663 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout:Upgraded: 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-19T18:46:11.664 INFO:teuthology.orchestra.run.smithi104.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:11.665 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:46:12.051 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-test 2024-02-19T18:46:12.636 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:30 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:46:12.802 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:46:12.803 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:46:12.803 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:46:12.803 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test x86_64 2:17.2.7-521.gc80e7168.el9 ceph 48 M 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite x86_64 2:17.2.7-521.gc80e7168.el9 ceph 163 k 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-02-19T18:46:12.804 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:12.805 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 49 M 2024-02-19T18:46:12.805 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 196 M 2024-02-19T18:46:12.805 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:46:13.020 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-02-19T18:46:13.053 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): libcephsqlite-17.2.7-521.gc80e7168.el9.x 654 kB/s | 163 kB 00:00 2024-02-19T18:46:13.079 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-02-19T18:46:13.137 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 186 kB 00:00 2024-02-19T18:46:13.162 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.5 MB/s | 217 kB 00:00 2024-02-19T18:46:17.737 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): ceph-test-17.2.7-521.gc80e7168.el9.x86_6 9.7 MB/s | 48 MB 00:04 2024-02-19T18:46:17.747 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:46:17.747 INFO:teuthology.orchestra.run.smithi104.stdout:Total 9.9 MB/s | 49 MB 00:04 2024-02-19T18:46:17.747 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:46:17.760 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:46:17.760 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:46:18.156 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:46:18.156 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:46:18.737 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:46:18.798 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-19T18:46:18.798 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2024-02-19T18:46:18.798 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2024-02-19T18:46:18.798 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2024-02-19T18:46:18.798 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2024-02-19T18:46:18.798 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2024-02-19T18:46:18.798 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /root 2024-02-19T18:46:18.799 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2024-02-19T18:46:18.799 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:18.802 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-19T18:46:18.837 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-19T18:46:18.880 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-19T18:46:18.929 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-19T18:46:18.957 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-19T18:46:20.107 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-19T18:46:20.179 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 24/26 2024-02-19T18:46:20.180 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:20.180 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-19T18:46:20.180 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:46:20.180 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-19T18:46:20.180 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:20.181 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-19T18:46:20.262 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-02-19T18:46:20.263 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-19T18:46:21.547 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 1/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 3/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 4/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 5/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 6/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 7/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x 8/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86 9/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 10/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 11/26 2024-02-19T18:46:21.548 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 12/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 13/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-02-19T18:46:21.549 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:17.2.7-521.gc80e7168.el9.x86_64 23/26 2024-02-19T18:46:21.552 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-02-19T18:46:21.552 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 25/26 2024-02-19T18:46:22.632 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-02-19T18:46:22.632 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:22.632 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-19T18:46:22.633 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:22.634 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:46:23.015 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-test 2024-02-19T18:46:23.397 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-19T18:46:23.472 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-19T18:46:23.586 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:28 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:46:23.751 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:17.2.7-521.gc80e7168.el9 ceph 48 M 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:17.2.7-521.gc80e7168.el9 ceph 163 k 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-19T18:46:23.753 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-19T18:46:23.754 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:23.754 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:46:23.754 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:23.754 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2024-02-19T18:46:23.754 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:23.754 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 49 M 2024-02-19T18:46:23.754 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 196 M 2024-02-19T18:46:23.754 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:46:23.932 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 300 kB 00:00 2024-02-19T18:46:23.966 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-02-19T18:46:24.091 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 186 kB 00:00 2024-02-19T18:46:24.150 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-02-19T18:46:24.175 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): libcephsqlite-17.2.7-521.gc80e7168.el9.x 387 kB/s | 163 kB 00:00 2024-02-19T18:46:25.824 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-19T18:46:25.824 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 1/6 2024-02-19T18:46:25.824 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2/6 2024-02-19T18:46:25.824 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-19T18:46:25.824 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-19T18:46:25.824 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-19T18:46:26.243 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-19T18:46:26.243 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:26.243 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:46:26.243 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:26.243 INFO:teuthology.orchestra.run.smithi104.stdout: jq-1.6-16.el9.x86_64 2024-02-19T18:46:26.243 INFO:teuthology.orchestra.run.smithi104.stdout: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:26.244 INFO:teuthology.orchestra.run.smithi104.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-19T18:46:26.244 INFO:teuthology.orchestra.run.smithi104.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-19T18:46:26.244 INFO:teuthology.orchestra.run.smithi104.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-19T18:46:26.244 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:26.244 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:46:26.555 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph 2024-02-19T18:46:27.161 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:00:45 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:46:27.334 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:46:27.336 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-02-19T18:46:27.336 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:46:27.336 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-02-19T18:46:27.336 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:46:27.336 INFO:teuthology.orchestra.run.smithi104.stdout: ceph x86_64 2:17.2.7-521.gc80e7168.el9 ceph 6.5 k 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.1 M 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr x86_64 2:17.2.7-521.gc80e7168.el9 ceph 1.4 M 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 239 k 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.7 M 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 15 M 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-19T18:46:27.337 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-19T18:46:27.338 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================== 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout:Install 33 Packages 2024-02-19T18:46:27.339 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:27.340 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 27 M 2024-02-19T18:46:27.340 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 99 M 2024-02-19T18:46:27.340 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:46:27.738 INFO:teuthology.orchestra.run.smithi104.stdout:(1/33): ceph-17.2.7-521.gc80e7168.el9.x86_64.rp 26 kB/s | 6.5 kB 00:00 2024-02-19T18:46:28.380 INFO:teuthology.orchestra.run.smithi104.stdout:(2/33): ceph-mon-17.2.7-521.gc80e7168.el9.x86_6 5.8 MB/s | 3.7 MB 00:00 2024-02-19T18:46:28.431 INFO:teuthology.orchestra.run.smithi104.stdout:(3/33): ceph-mds-17.2.7-521.gc80e7168.el9.x86_6 2.2 MB/s | 2.1 MB 00:00 2024-02-19T18:46:28.480 INFO:teuthology.orchestra.run.smithi104.stdout:(4/33): ceph-mgr-17.2.7-521.gc80e7168.el9.x86_6 1.4 MB/s | 1.4 MB 00:00 2024-02-19T18:46:28.539 INFO:teuthology.orchestra.run.smithi104.stdout:(5/33): ceph-mgr-modules-core-17.2.7-521.gc80e7 2.2 MB/s | 239 kB 00:00 2024-02-19T18:46:28.608 INFO:teuthology.orchestra.run.smithi104.stdout:(6/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-02-19T18:46:28.631 INFO:teuthology.orchestra.run.smithi104.stdout:(7/33): python3-markupsafe-1.1.1-12.el9.x86_64. 347 kB/s | 32 kB 00:00 2024-02-19T18:46:28.657 INFO:teuthology.orchestra.run.smithi104.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-02-19T18:46:28.685 INFO:teuthology.orchestra.run.smithi104.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-02-19T18:46:28.707 INFO:teuthology.orchestra.run.smithi104.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-19T18:46:28.757 INFO:teuthology.orchestra.run.smithi104.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-02-19T18:46:28.783 INFO:teuthology.orchestra.run.smithi104.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-02-19T18:46:28.791 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): ceph-test-17.2.7-521.gc80e7168.el9.x86_6 9.5 MB/s | 48 MB 00:05 2024-02-19T18:46:28.799 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:46:28.799 INFO:teuthology.orchestra.run.smithi077.stdout:Total 9.7 MB/s | 49 MB 00:05 2024-02-19T18:46:28.799 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:46:28.808 INFO:teuthology.orchestra.run.smithi104.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-02-19T18:46:28.817 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:46:28.817 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:46:28.833 INFO:teuthology.orchestra.run.smithi104.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2024-02-19T18:46:28.858 INFO:teuthology.orchestra.run.smithi104.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-19T18:46:29.211 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:46:29.212 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:46:29.828 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:46:29.909 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 1/6 2024-02-19T18:46:29.951 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.6-16.el9.x86_64 2/6 2024-02-19T18:46:30.001 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 3/6 2024-02-19T18:46:30.058 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 4/6 2024-02-19T18:46:30.085 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-19T18:46:30.117 INFO:teuthology.orchestra.run.smithi104.stdout:(16/33): ceph-osd-17.2.7-521.gc80e7168.el9.x86_ 8.8 MB/s | 15 MB 00:01 2024-02-19T18:46:34.512 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 5/6 2024-02-19T18:46:34.585 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-19T18:46:36.872 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 6/6 2024-02-19T18:46:36.872 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 1/6 2024-02-19T18:46:36.872 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2/6 2024-02-19T18:46:36.872 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/6 2024-02-19T18:46:36.872 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/6 2024-02-19T18:46:36.872 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.6-16.el9.x86_64 5/6 2024-02-19T18:46:37.268 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 6/6 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout: jq-1.6-16.el9.x86_64 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:37.269 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:46:37.561 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph 2024-02-19T18:46:38.166 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:43 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:46:38.345 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================== 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================== 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:17.2.7-521.gc80e7168.el9 ceph 6.5 k 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.1 M 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:17.2.7-521.gc80e7168.el9 ceph 1.4 M 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 239 k 2024-02-19T18:46:38.347 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:17.2.7-521.gc80e7168.el9 ceph 3.7 M 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 15 M 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-19T18:46:38.348 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-19T18:46:38.349 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-19T18:46:38.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-19T18:46:38.350 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:38.350 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:46:38.350 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================== 2024-02-19T18:46:38.350 INFO:teuthology.orchestra.run.smithi077.stdout:Install 33 Packages 2024-02-19T18:46:38.350 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:38.350 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 27 M 2024-02-19T18:46:38.351 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 99 M 2024-02-19T18:46:38.351 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:46:38.812 INFO:teuthology.orchestra.run.smithi077.stdout:(1/33): ceph-17.2.7-521.gc80e7168.el9.x86_64.rp 21 kB/s | 6.5 kB 00:00 2024-02-19T18:46:39.319 INFO:teuthology.orchestra.run.smithi077.stdout:(2/33): ceph-mgr-17.2.7-521.gc80e7168.el9.x86_6 1.7 MB/s | 1.4 MB 00:00 2024-02-19T18:46:40.176 INFO:teuthology.orchestra.run.smithi077.stdout:(3/33): ceph-mds-17.2.7-521.gc80e7168.el9.x86_6 1.2 MB/s | 2.1 MB 00:01 2024-02-19T18:46:40.376 INFO:teuthology.orchestra.run.smithi077.stdout:(4/33): ceph-mon-17.2.7-521.gc80e7168.el9.x86_6 2.4 MB/s | 3.7 MB 00:01 2024-02-19T18:46:40.402 INFO:teuthology.orchestra.run.smithi077.stdout:(5/33): ceph-mgr-modules-core-17.2.7-521.gc80e7 1.0 MB/s | 239 kB 00:00 2024-02-19T18:46:40.560 INFO:teuthology.orchestra.run.smithi077.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 200 kB/s | 32 kB 00:00 2024-02-19T18:46:40.585 INFO:teuthology.orchestra.run.smithi077.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 691 kB/s | 144 kB 00:00 2024-02-19T18:46:40.602 INFO:teuthology.orchestra.run.smithi077.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-02-19T18:46:40.619 INFO:teuthology.orchestra.run.smithi077.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-02-19T18:46:40.661 INFO:teuthology.orchestra.run.smithi077.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-02-19T18:46:40.720 INFO:teuthology.orchestra.run.smithi077.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-02-19T18:46:40.745 INFO:teuthology.orchestra.run.smithi077.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-02-19T18:46:40.762 INFO:teuthology.orchestra.run.smithi077.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-02-19T18:46:40.787 INFO:teuthology.orchestra.run.smithi077.stdout:(14/33): python3-requests-2.25.1-8.el9.noarch.r 2.6 MB/s | 113 kB 00:00 2024-02-19T18:46:40.813 INFO:teuthology.orchestra.run.smithi077.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-02-19T18:46:41.113 INFO:teuthology.orchestra.run.smithi077.stdout:(16/33): ceph-osd-17.2.7-521.gc80e7168.el9.x86_ 8.5 MB/s | 15 MB 00:01 2024-02-19T18:46:41.230 INFO:teuthology.orchestra.run.smithi077.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 98 kB/s | 43 kB 00:00 2024-02-19T18:46:41.347 INFO:teuthology.orchestra.run.smithi077.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 322 kB/s | 172 kB 00:00 2024-02-19T18:46:41.372 INFO:teuthology.orchestra.run.smithi077.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 75 kB/s | 11 kB 00:00 2024-02-19T18:46:41.456 INFO:teuthology.orchestra.run.smithi077.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-02-19T18:46:41.481 INFO:teuthology.orchestra.run.smithi077.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-02-19T18:46:41.564 INFO:teuthology.orchestra.run.smithi077.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-02-19T18:46:41.590 INFO:teuthology.orchestra.run.smithi077.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2024-02-19T18:46:41.690 INFO:teuthology.orchestra.run.smithi077.stdout:(24/33): python3-cherrypy-18.6.1-2.el9.noarch.r 622 kB/s | 358 kB 00:00 2024-02-19T18:46:41.715 INFO:teuthology.orchestra.run.smithi077.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 307 kB/s | 46 kB 00:00 2024-02-19T18:46:41.741 INFO:teuthology.orchestra.run.smithi077.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 524 kB/s | 79 kB 00:00 2024-02-19T18:46:41.841 INFO:teuthology.orchestra.run.smithi077.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-02-19T18:46:41.866 INFO:teuthology.orchestra.run.smithi077.stdout:(28/33): python3-portend-3.1.0-2.el9.noarch.rpm 109 kB/s | 16 kB 00:00 2024-02-19T18:46:41.883 INFO:teuthology.orchestra.run.smithi077.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 633 kB/s | 90 kB 00:00 2024-02-19T18:46:41.950 INFO:teuthology.orchestra.run.smithi077.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 329 kB/s | 36 kB 00:00 2024-02-19T18:46:42.050 INFO:teuthology.orchestra.run.smithi077.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-02-19T18:46:42.075 INFO:teuthology.orchestra.run.smithi077.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.1 MB/s | 230 kB 00:00 2024-02-19T18:46:42.184 INFO:teuthology.orchestra.run.smithi077.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.4 MB/s | 427 kB 00:00 2024-02-19T18:46:42.188 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:46:42.189 INFO:teuthology.orchestra.run.smithi077.stdout:Total 7.0 MB/s | 27 MB 00:03 2024-02-19T18:46:42.189 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:46:42.227 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:46:42.228 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:46:42.562 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:46:42.562 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:46:43.794 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:46:43.864 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-19T18:46:43.886 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-19T18:46:43.919 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-19T18:46:44.010 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-19T18:46:44.089 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-19T18:46:44.130 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-19T18:46:44.201 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-19T18:46:44.255 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-19T18:46:44.293 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-19T18:46:44.343 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-19T18:46:44.398 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-19T18:46:44.582 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-19T18:46:44.622 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-19T18:46:44.653 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-19T18:46:44.688 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-19T18:46:44.720 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-19T18:46:44.795 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-19T18:46:44.830 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:46:44.863 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-19T18:46:44.904 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-19T18:46:44.938 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-19T18:46:44.973 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-19T18:46:45.062 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-19T18:46:45.107 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-19T18:46:45.185 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-19T18:46:45.321 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-19T18:46:45.451 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:46:45.602 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 28/33 2024-02-19T18:46:45.634 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-19T18:46:45.660 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-19T18:46:45.661 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:45.661 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-19T18:46:45.661 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:46:45.661 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:46:45.661 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:46.996 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-19T18:46:47.019 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-19T18:46:47.020 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:47.020 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-19T18:46:47.020 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:46:47.020 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:46:47.020 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:47.394 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-19T18:46:47.416 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-19T18:46:47.417 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:47.417 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-19T18:46:47.417 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:46:47.417 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:46:47.417 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:47.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-19T18:46:47.677 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-19T18:46:47.677 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:47.677 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-19T18:46:47.677 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:46:47.677 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:46:47.677 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:47.708 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-19T18:46:48.314 INFO:teuthology.orchestra.run.smithi104.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.2 kB/s | 43 kB 00:19 2024-02-19T18:46:48.456 INFO:teuthology.orchestra.run.smithi104.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 75 kB/s | 11 kB 00:00 2024-02-19T18:46:48.581 INFO:teuthology.orchestra.run.smithi104.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 141 kB/s | 18 kB 00:00 2024-02-19T18:46:48.723 INFO:teuthology.orchestra.run.smithi104.stdout:(20/33): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2024-02-19T18:46:48.840 INFO:teuthology.orchestra.run.smithi104.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 167 kB/s | 19 kB 00:00 2024-02-19T18:46:49.007 INFO:teuthology.orchestra.run.smithi104.stdout:(22/33): python3-jaraco-text-3.2.0-6.el9.noarch 118 kB/s | 20 kB 00:00 2024-02-19T18:46:49.148 INFO:teuthology.orchestra.run.smithi104.stdout:(23/33): python3-logutils-0.3.5-21.el9.noarch.r 327 kB/s | 46 kB 00:00 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 1/33 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2/33 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 3/33 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 4/33 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 5/33 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 6/33 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-19T18:46:49.304 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-19T18:46:49.305 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-19T18:46:49.306 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-19T18:46:49.307 INFO:teuthology.orchestra.run.smithi104.stdout:(24/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 8.4 kB/s | 172 kB 00:20 2024-02-19T18:46:49.333 INFO:teuthology.orchestra.run.smithi104.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 429 kB/s | 79 kB 00:00 2024-02-19T18:46:49.466 INFO:teuthology.orchestra.run.smithi104.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 123 kB/s | 16 kB 00:00 2024-02-19T18:46:49.508 INFO:teuthology.orchestra.run.smithi104.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-02-19T18:46:49.608 INFO:teuthology.orchestra.run.smithi104.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 634 kB/s | 90 kB 00:00 2024-02-19T18:46:49.625 INFO:teuthology.orchestra.run.smithi104.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 305 kB/s | 36 kB 00:00 2024-02-19T18:46:49.792 INFO:teuthology.orchestra.run.smithi104.stdout:(30/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-02-19T18:46:49.826 INFO:teuthology.orchestra.run.smithi104.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-02-19T18:46:49.929 INFO:teuthology.orchestra.run.smithi104.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 149 kB/s | 20 kB 00:00 2024-02-19T18:46:50.093 INFO:teuthology.orchestra.run.smithi104.stdout:(33/33): python3-cherrypy-18.6.1-2.el9.noarch.r 18 kB/s | 358 kB 00:19 2024-02-19T18:46:50.096 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:46:50.097 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.2 MB/s | 27 MB 00:22 2024-02-19T18:46:50.097 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:46:50.136 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:46:50.136 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:46:50.349 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-19T18:46:50.349 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:50.349 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:46:50.349 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:50.349 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:50.349 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-19T18:46:50.350 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ply-3.11-14.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-19T18:46:50.351 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-19T18:46:50.352 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-19T18:46:50.352 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:50.352 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:46:50.464 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:46:50.465 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:46:50.680 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-base 2024-02-19T18:46:51.260 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:56 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:46:51.373 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:46:51.423 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:46:51.424 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:46:51.424 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:46:51.485 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install cephadm 2024-02-19T18:46:51.676 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:46:51.745 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-02-19T18:46:51.786 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-02-19T18:46:51.819 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-02-19T18:46:51.910 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-02-19T18:46:51.980 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-02-19T18:46:52.030 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-02-19T18:46:52.062 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:57 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:46:52.100 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-02-19T18:46:52.154 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-02-19T18:46:52.193 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-02-19T18:46:52.229 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:46:52.229 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:52.229 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:46:52.229 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:52.229 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:46:52.229 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 80 k 2024-02-19T18:46:52.230 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:52.230 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:46:52.230 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:52.230 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-19T18:46:52.230 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:52.230 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 80 k 2024-02-19T18:46:52.230 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 364 k 2024-02-19T18:46:52.230 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:46:52.251 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-02-19T18:46:52.305 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-02-19T18:46:52.444 INFO:teuthology.orchestra.run.smithi077.stdout:cephadm-17.2.7-521.gc80e7168.el9.noarch.rpm 379 kB/s | 80 kB 00:00 2024-02-19T18:46:52.444 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:46:52.444 INFO:teuthology.orchestra.run.smithi077.stdout:Total 377 kB/s | 80 kB 00:00 2024-02-19T18:46:52.444 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:46:52.450 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:46:52.450 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:46:52.450 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:46:52.450 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:46:52.489 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-02-19T18:46:52.522 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:46:52.550 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-02-19T18:46:52.580 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-02-19T18:46:52.615 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-02-19T18:46:52.645 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-02-19T18:46:52.720 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-02-19T18:46:52.755 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:46:52.788 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-02-19T18:46:52.828 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-02-19T18:46:52.863 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-02-19T18:46:52.898 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-02-19T18:46:52.973 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-19T18:46:53.004 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-02-19T18:46:53.053 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-02-19T18:46:53.054 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-19T18:46:53.125 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-02-19T18:46:53.262 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-02-19T18:46:53.386 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:46:53.434 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-19T18:46:53.538 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 28/33 2024-02-19T18:46:53.567 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-19T18:46:53.592 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 29/33 2024-02-19T18:46:53.592 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:53.592 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-19T18:46:53.592 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:46:53.593 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-19T18:46:53.593 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:53.741 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-19T18:46:53.741 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:53.741 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:46:53.742 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:46:53.742 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:53.742 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:46:53.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T18:46:54.534 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:59 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:46:54.703 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:46:54.704 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:54.704 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2024-02-19T18:46:54.704 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:54.704 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:46:54.704 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-521.gc80e7168.el9 ceph 137 k 2024-02-19T18:46:54.704 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:54.704 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:46:54.704 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:54.705 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-19T18:46:54.705 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:54.705 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 137 k 2024-02-19T18:46:54.705 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 379 k 2024-02-19T18:46:54.705 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:46:54.909 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-19T18:46:54.934 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 30/33 2024-02-19T18:46:54.935 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:54.935 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-19T18:46:54.935 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:46:54.935 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-19T18:46:54.935 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:55.073 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-immutable-object-cache-17.2.7-521.gc80e716 373 kB/s | 137 kB 00:00 2024-02-19T18:46:55.073 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:46:55.073 INFO:teuthology.orchestra.run.smithi077.stdout:Total 371 kB/s | 137 kB 00:00 2024-02-19T18:46:55.073 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:46:55.081 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:46:55.081 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:46:55.132 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:46:55.132 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:46:55.285 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:46:55.310 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-19T18:46:55.332 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 31/33 2024-02-19T18:46:55.332 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:55.333 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-19T18:46:55.333 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:46:55.333 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-19T18:46:55.333 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:55.357 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-19T18:46:55.380 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-19T18:46:55.381 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:55.381 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-19T18:46:55.381 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:55.535 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-19T18:46:55.558 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 32/33 2024-02-19T18:46:55.558 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:46:55.558 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-19T18:46:55.558 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:46:55.558 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-19T18:46:55.559 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:55.615 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-19T18:46:56.360 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-19T18:46:56.360 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:56.360 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:46:56.360 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:56.360 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:56.360 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:46:56.604 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr 2024-02-19T18:46:57.042 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 33/33 2024-02-19T18:46:57.042 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-2:17.2.7-521.gc80e7168.el9.x86_64 1/33 2024-02-19T18:46:57.042 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 3/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 4/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 5/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.n 6/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-02-19T18:46:57.043 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 15/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-02-19T18:46:57.044 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-02-19T18:46:57.045 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-02-19T18:46:57.045 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-02-19T18:46:57.046 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-02-19T18:46:57.046 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-02-19T18:46:57.046 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-02-19T18:46:57.046 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-02-19T18:46:57.046 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-02-19T18:46:57.046 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-02-19T18:46:57.046 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-02-19T18:46:57.188 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:02 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:46:57.302 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:46:57.352 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:46:57.353 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:46:57.354 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:46:57.419 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T18:46:57.792 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mds-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-modules-core-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mon-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-osd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-19T18:46:57.793 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-ply-3.11-14.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-19T18:46:57.794 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:57.795 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:46:57.994 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:02 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:46:58.160 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:46:58.160 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 1.7 M 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 23 k 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 15 k 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:46:58.161 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:46:58.162 INFO:teuthology.orchestra.run.smithi077.stdout:Install 7 Packages 2024-02-19T18:46:58.162 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:46:58.162 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 2.0 M 2024-02-19T18:46:58.162 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 8.1 M 2024-02-19T18:46:58.162 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:46:58.163 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-base 2024-02-19T18:46:58.668 INFO:teuthology.orchestra.run.smithi077.stdout:(1/7): ceph-grafana-dashboards-17.2.7-521.gc80e 68 kB/s | 23 kB 00:00 2024-02-19T18:46:58.750 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:16 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:46:58.785 INFO:teuthology.orchestra.run.smithi077.stdout:(2/7): ceph-prometheus-alerts-17.2.7-521.gc80e7 32 kB/s | 15 kB 00:00 2024-02-19T18:46:58.852 INFO:teuthology.orchestra.run.smithi077.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 49 kB/s | 9.0 kB 00:00 2024-02-19T18:46:58.864 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-base-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:46:58.915 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:46:58.916 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:46:58.916 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:46:58.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install cephadm 2024-02-19T18:46:59.044 INFO:teuthology.orchestra.run.smithi077.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 160 kB/s | 31 kB 00:00 2024-02-19T18:46:59.094 INFO:teuthology.orchestra.run.smithi077.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 132 kB/s | 41 kB 00:00 2024-02-19T18:46:59.436 INFO:teuthology.orchestra.run.smithi077.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 481 kB/s | 188 kB 00:00 2024-02-19T18:46:59.568 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:17 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:46:59.730 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:46:59.731 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:46:59.731 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:46:59.731 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 80 k 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 80 k 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 364 k 2024-02-19T18:46:59.732 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:46:59.942 INFO:teuthology.orchestra.run.smithi104.stdout:cephadm-17.2.7-521.gc80e7168.el9.noarch.rpm 384 kB/s | 80 kB 00:00 2024-02-19T18:46:59.942 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:46:59.943 INFO:teuthology.orchestra.run.smithi104.stdout:Total 381 kB/s | 80 kB 00:00 2024-02-19T18:46:59.943 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:46:59.948 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:46:59.948 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:46:59.950 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:46:59.950 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:00.024 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:00.381 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-19T18:47:00.463 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-19T18:47:00.828 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-19T18:47:00.944 INFO:teuthology.orchestra.run.smithi077.stdout:(7/7): ceph-mgr-dashboard-17.2.7-521.gc80e7168. 663 kB/s | 1.7 MB 00:02 2024-02-19T18:47:00.946 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:00.946 INFO:teuthology.orchestra.run.smithi077.stdout:Total 734 kB/s | 2.0 MB 00:02 2024-02-19T18:47:00.947 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:00.956 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:00.956 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:01.028 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:01.029 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/1 2024-02-19T18:47:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:01.128 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:01.128 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:01.128 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:01.329 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:01.339 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-immutable-object-cache 2024-02-19T18:47:01.396 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-19T18:47:01.438 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-19T18:47:01.484 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-19T18:47:01.516 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-19T18:47:01.545 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 5/7 2024-02-19T18:47:01.772 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 6/7 2024-02-19T18:47:01.798 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-19T18:47:01.929 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:19 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:02.095 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:02.096 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:02.096 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-02-19T18:47:02.096 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:02.096 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:02.096 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-521.gc80e7168.el9 ceph 137 k 2024-02-19T18:47:02.097 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:02.097 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:02.097 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:02.097 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-19T18:47:02.097 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:02.097 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 137 k 2024-02-19T18:47:02.097 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 379 k 2024-02-19T18:47:02.097 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:02.322 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-19T18:47:02.322 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 1/7 2024-02-19T18:47:02.322 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2/7 2024-02-19T18:47:02.322 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 3/7 2024-02-19T18:47:02.323 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-19T18:47:02.323 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-19T18:47:02.323 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-19T18:47:02.446 INFO:teuthology.orchestra.run.smithi104.stdout:ceph-immutable-object-cache-17.2.7-521.gc80e716 393 kB/s | 137 kB 00:00 2024-02-19T18:47:02.446 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:02.447 INFO:teuthology.orchestra.run.smithi104.stdout:Total 391 kB/s | 137 kB 00:00 2024-02-19T18:47:02.447 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:02.454 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:02.454 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:02.506 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:02.506 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:02.622 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-19T18:47:02.622 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:02.622 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:02.623 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:02.664 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:02.714 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-19T18:47:02.738 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-19T18:47:02.738 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:47:02.738 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-19T18:47:02.738 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:02.841 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T18:47:03.413 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:08 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:03.580 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:03.581 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================== 2024-02-19T18:47:03.581 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:47:03.581 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================== 2024-02-19T18:47:03.581 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:03.581 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 7.4 M 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-19T18:47:03.582 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:03.583 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:03.583 INFO:teuthology.orchestra.run.smithi077.stdout:=========================================================================================== 2024-02-19T18:47:03.583 INFO:teuthology.orchestra.run.smithi077.stdout:Install 12 Packages 2024-02-19T18:47:03.583 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:03.583 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 37 M 2024-02-19T18:47:03.583 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 209 M 2024-02-19T18:47:03.584 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.e 1/1 2024-02-19T18:47:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:03.736 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-immutable-object-cache-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:03.736 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:03.736 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:03.826 INFO:teuthology.orchestra.run.smithi077.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 250 kB/s | 32 kB 00:00 2024-02-19T18:47:03.860 INFO:teuthology.orchestra.run.smithi077.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-02-19T18:47:03.893 INFO:teuthology.orchestra.run.smithi077.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-02-19T18:47:03.978 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr 2024-02-19T18:47:04.028 INFO:teuthology.orchestra.run.smithi077.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.1 MB/s | 3.0 MB 00:00 2024-02-19T18:47:04.086 INFO:teuthology.orchestra.run.smithi077.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-02-19T18:47:04.160 INFO:teuthology.orchestra.run.smithi077.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-02-19T18:47:04.186 INFO:teuthology.orchestra.run.smithi077.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-02-19T18:47:04.295 INFO:teuthology.orchestra.run.smithi077.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 7.4 MB/s | 802 kB 00:00 2024-02-19T18:47:04.495 INFO:teuthology.orchestra.run.smithi077.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-02-19T18:47:04.520 INFO:teuthology.orchestra.run.smithi077.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 871 kB/s | 194 kB 00:00 2024-02-19T18:47:04.570 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:22 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:04.682 INFO:teuthology.orchestra.run.smithi104.stdout:Package ceph-mgr-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:04.687 INFO:teuthology.orchestra.run.smithi077.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-02-19T18:47:04.733 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:04.734 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:47:04.734 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:04.808 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-dashboard 2024-02-19T18:47:05.386 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:23 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:05.552 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:05.552 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:05.552 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:47:05.552 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 1.7 M 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 23 k 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 15 k 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout:Installing weak dependencies: 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:05.553 INFO:teuthology.orchestra.run.smithi104.stdout:Install 7 Packages 2024-02-19T18:47:05.554 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:05.554 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.0 M 2024-02-19T18:47:05.554 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 8.1 M 2024-02-19T18:47:05.554 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:05.932 INFO:teuthology.orchestra.run.smithi104.stdout:(1/7): ceph-grafana-dashboards-17.2.7-521.gc80e 124 kB/s | 23 kB 00:00 2024-02-19T18:47:05.987 INFO:teuthology.orchestra.run.smithi077.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 3.2 MB/s | 7.4 MB 00:02 2024-02-19T18:47:05.990 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:05.990 INFO:teuthology.orchestra.run.smithi077.stdout:Total 15 MB/s | 37 MB 00:02 2024-02-19T18:47:06.013 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:06.058 INFO:teuthology.orchestra.run.smithi104.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 73 kB/s | 9.0 kB 00:00 2024-02-19T18:47:06.108 INFO:teuthology.orchestra.run.smithi104.stdout:(3/7): ceph-prometheus-alerts-17.2.7-521.gc80e7 40 kB/s | 15 kB 00:00 2024-02-19T18:47:06.167 INFO:teuthology.orchestra.run.smithi104.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 377 kB/s | 41 kB 00:00 2024-02-19T18:47:06.292 INFO:teuthology.orchestra.run.smithi104.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 167 kB/s | 31 kB 00:00 2024-02-19T18:47:06.355 INFO:teuthology.orchestra.run.smithi104.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2024-02-19T18:47:06.467 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:06.467 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:06.550 INFO:teuthology.orchestra.run.smithi104.stdout:(7/7): ceph-mgr-dashboard-17.2.7-521.gc80e7168. 2.1 MB/s | 1.7 MB 00:00 2024-02-19T18:47:06.552 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:06.552 INFO:teuthology.orchestra.run.smithi104.stdout:Total 2.0 MB/s | 2.0 MB 00:00 2024-02-19T18:47:06.553 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:06.561 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:06.561 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:06.628 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:06.628 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:06.920 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:06.985 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-02-19T18:47:07.019 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-02-19T18:47:07.066 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-02-19T18:47:07.097 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-02-19T18:47:07.129 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 5/7 2024-02-19T18:47:07.162 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:07.242 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-19T18:47:07.274 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-19T18:47:07.345 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 6/7 2024-02-19T18:47:07.369 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:47:07.379 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-19T18:47:07.411 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-19T18:47:07.588 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:47:07.619 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:47:07.701 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-19T18:47:07.946 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 7/7 2024-02-19T18:47:07.946 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.n 1/7 2024-02-19T18:47:07.946 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2/7 2024-02-19T18:47:07.946 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.no 3/7 2024-02-19T18:47:07.946 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-02-19T18:47:07.946 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-02-19T18:47:07.946 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-02-19T18:47:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-02-19T18:47:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:08.260 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-grafana-dashboards-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-dashboard-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-prometheus-alerts-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-02-19T18:47:08.260 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-02-19T18:47:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-19T18:47:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-19T18:47:08.261 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:08.261 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:08.360 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-19T18:47:08.472 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-19T18:47:08.496 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-diskprediction-local 2024-02-19T18:47:09.068 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:27 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:09.243 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:09.244 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================== 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================== 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 7.4 M 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-19T18:47:09.245 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout:=========================================================================================== 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout:Install 12 Packages 2024-02-19T18:47:09.246 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:09.247 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 37 M 2024-02-19T18:47:09.247 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 209 M 2024-02-19T18:47:09.247 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:09.498 INFO:teuthology.orchestra.run.smithi104.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2024-02-19T18:47:09.540 INFO:teuthology.orchestra.run.smithi104.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 427 kB/s | 17 kB 00:00 2024-02-19T18:47:09.599 INFO:teuthology.orchestra.run.smithi104.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 602 kB/s | 34 kB 00:00 2024-02-19T18:47:09.666 INFO:teuthology.orchestra.run.smithi104.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.8 MB/s | 3.0 MB 00:00 2024-02-19T18:47:09.711 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-19T18:47:09.999 INFO:teuthology.orchestra.run.smithi104.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-02-19T18:47:10.091 INFO:teuthology.orchestra.run.smithi104.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-02-19T18:47:10.116 INFO:teuthology.orchestra.run.smithi104.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 217 kB 00:00 2024-02-19T18:47:10.517 INFO:teuthology.orchestra.run.smithi104.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 802 kB 00:00 2024-02-19T18:47:10.738 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-19T18:47:10.776 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-19T18:47:10.850 INFO:teuthology.orchestra.run.smithi104.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 21 MB/s | 16 MB 00:00 2024-02-19T18:47:10.878 INFO:teuthology.orchestra.run.smithi104.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 544 kB/s | 194 kB 00:00 2024-02-19T18:47:11.175 INFO:teuthology.orchestra.run.smithi104.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 756 kB/s | 245 kB 00:00 2024-02-19T18:47:13.339 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-19T18:47:13.339 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 1/12 2024-02-19T18:47:13.339 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-19T18:47:13.339 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:47:13.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-19T18:47:13.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:47:13.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:47:13.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-19T18:47:13.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-19T18:47:13.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-19T18:47:13.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-19T18:47:13.340 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-19T18:47:13.769 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-19T18:47:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-19T18:47:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-19T18:47:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-19T18:47:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-19T18:47:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-19T18:47:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-19T18:47:13.770 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:13.770 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:14.137 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-rook 2024-02-19T18:47:14.728 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:19 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:14.896 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:14.897 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-02-19T18:47:14.897 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:47:14.897 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-02-19T18:47:14.897 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:14.897 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 47 k 2024-02-19T18:47:14.897 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:47:14.897 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-19T18:47:14.897 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:14.898 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-02-19T18:47:14.899 INFO:teuthology.orchestra.run.smithi077.stdout:Install 13 Packages 2024-02-19T18:47:14.899 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:14.899 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 2.0 M 2024-02-19T18:47:14.899 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 26 M 2024-02-19T18:47:14.899 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:15.068 INFO:teuthology.orchestra.run.smithi077.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 18 kB 00:00 2024-02-19T18:47:15.093 INFO:teuthology.orchestra.run.smithi077.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 166 kB/s | 24 kB 00:00 2024-02-19T18:47:15.143 INFO:teuthology.orchestra.run.smithi077.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-02-19T18:47:15.168 INFO:teuthology.orchestra.run.smithi077.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-19T18:47:15.194 INFO:teuthology.orchestra.run.smithi077.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-02-19T18:47:15.219 INFO:teuthology.orchestra.run.smithi077.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2024-02-19T18:47:15.353 INFO:teuthology.orchestra.run.smithi077.stdout:(7/13): ceph-mgr-rook-17.2.7-521.gc80e7168.el9. 117 kB/s | 47 kB 00:00 2024-02-19T18:47:15.478 INFO:teuthology.orchestra.run.smithi077.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-02-19T18:47:15.503 INFO:teuthology.orchestra.run.smithi077.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 53 kB/s | 15 kB 00:00 2024-02-19T18:47:15.595 INFO:teuthology.orchestra.run.smithi077.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 840 kB/s | 203 kB 00:00 2024-02-19T18:47:15.704 INFO:teuthology.orchestra.run.smithi077.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 4.5 MB/s | 1.0 MB 00:00 2024-02-19T18:47:15.737 INFO:teuthology.orchestra.run.smithi077.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-02-19T18:47:15.754 INFO:teuthology.orchestra.run.smithi077.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-02-19T18:47:15.757 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:15.757 INFO:teuthology.orchestra.run.smithi077.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-02-19T18:47:15.757 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:15.770 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:15.770 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:15.869 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:15.869 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:16.390 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:16.561 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-19T18:47:16.613 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-19T18:47:16.677 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-19T18:47:16.715 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-19T18:47:16.756 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-19T18:47:16.860 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-19T18:47:16.958 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-19T18:47:17.007 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-19T18:47:17.727 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-19T18:47:17.779 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-19T18:47:17.813 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-19T18:47:17.859 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-19T18:47:17.883 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 1/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-19T18:47:18.678 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-19T18:47:18.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-19T18:47:18.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-19T18:47:18.679 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-19T18:47:19.038 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-19T18:47:19.039 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-19T18:47:19.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-19T18:47:19.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-19T18:47:19.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-19T18:47:19.040 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:19.040 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:19.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T18:47:19.391 INFO:teuthology.orchestra.run.smithi104.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-5 755 kB/s | 7.4 MB 00:10 2024-02-19T18:47:19.394 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:19.395 INFO:teuthology.orchestra.run.smithi104.stdout:Total 3.7 MB/s | 37 MB 00:10 2024-02-19T18:47:19.417 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:19.444 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:19.445 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:19.862 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:24 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:19.869 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:19.870 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:20.027 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:20.027 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-02-19T18:47:20.027 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:47:20.027 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 121 k 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2024-02-19T18:47:20.028 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:20.029 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 6.8 M 2024-02-19T18:47:20.029 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 33 M 2024-02-19T18:47:20.029 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:20.530 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): ceph-mgr-cephadm-17.2.7-521.gc80e7168.el 450 kB/s | 121 kB 00:00 2024-02-19T18:47:20.562 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:20.648 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 588 kB/s | 227 kB 00:00 2024-02-19T18:47:20.648 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-02-19T18:47:20.678 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-02-19T18:47:20.780 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:47:20.832 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-02-19T18:47:20.931 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.6 MB/s | 5.8 MB 00:00 2024-02-19T18:47:20.956 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 187 kB/s | 58 kB 00:00 2024-02-19T18:47:21.010 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:47:21.031 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:47:21.065 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-02-19T18:47:21.090 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-02-19T18:47:21.094 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:21.094 INFO:teuthology.orchestra.run.smithi077.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2024-02-19T18:47:21.094 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:21.105 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:21.105 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:21.112 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-02-19T18:47:21.192 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:21.192 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:21.488 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:21.646 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-19T18:47:21.708 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-19T18:47:21.764 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-02-19T18:47:21.876 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-02-19T18:47:22.224 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-19T18:47:22.313 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-19T18:47:22.374 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-19T18:47:22.404 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-19T18:47:23.121 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-02-19T18:47:23.269 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-19T18:47:23.270 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/6 2024-02-19T18:47:23.270 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-19T18:47:23.270 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-19T18:47:23.270 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-19T18:47:23.270 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-19T18:47:23.569 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-19T18:47:23.569 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:23.569 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:23.569 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:23.569 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-19T18:47:23.569 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-19T18:47:23.569 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-19T18:47:23.569 INFO:teuthology.orchestra.run.smithi077.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-19T18:47:23.570 INFO:teuthology.orchestra.run.smithi077.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-19T18:47:23.570 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:23.570 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:23.828 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-fuse 2024-02-19T18:47:24.160 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-02-19T18:47:24.204 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-19T18:47:24.406 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:29 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:24.573 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:24.573 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 812 k 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout:Install 2 Packages 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:24.574 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 891 k 2024-02-19T18:47:24.575 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 2.6 M 2024-02-19T18:47:24.575 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:24.792 INFO:teuthology.orchestra.run.smithi077.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 362 kB/s | 79 kB 00:00 2024-02-19T18:47:25.625 INFO:teuthology.orchestra.run.smithi077.stdout:(2/2): ceph-fuse-17.2.7-521.gc80e7168.el9.x86_6 773 kB/s | 812 kB 00:01 2024-02-19T18:47:25.626 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:25.626 INFO:teuthology.orchestra.run.smithi077.stdout:Total 846 kB/s | 891 kB 00:01 2024-02-19T18:47:25.627 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:25.647 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:25.647 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:25.722 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:25.723 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:25.902 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:26.028 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-19T18:47:26.107 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-19T18:47:26.684 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 12/12 2024-02-19T18:47:26.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e71 1/12 2024-02-19T18:47:26.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-02-19T18:47:26.684 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-02-19T18:47:26.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-02-19T18:47:26.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-02-19T18:47:26.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-02-19T18:47:26.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-02-19T18:47:26.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-02-19T18:47:26.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-02-19T18:47:26.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-02-19T18:47:26.685 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-02-19T18:47:26.793 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-19T18:47:26.793 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/2 2024-02-19T18:47:27.108 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-19T18:47:27.108 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:27.108 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:27.108 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-19T18:47:27.108 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:27.108 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:27.168 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-02-19T18:47:27.168 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:27.168 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:27.168 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-diskprediction-local-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:27.168 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-19T18:47:27.168 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:27.169 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:27.396 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados-devel 2024-02-19T18:47:27.539 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-rook 2024-02-19T18:47:27.993 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:32 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:28.142 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:46 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:28.164 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:28.164 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:28.164 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2024-02-19T18:47:28.164 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:28.164 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:28.164 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 126 k 2024-02-19T18:47:28.164 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:28.164 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:28.165 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:28.165 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-19T18:47:28.165 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:28.165 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 126 k 2024-02-19T18:47:28.165 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 448 k 2024-02-19T18:47:28.165 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:28.321 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 47 k 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-19T18:47:28.322 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout:Install 13 Packages 2024-02-19T18:47:28.323 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:28.324 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.0 M 2024-02-19T18:47:28.324 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 26 M 2024-02-19T18:47:28.324 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:28.621 INFO:teuthology.orchestra.run.smithi077.stdout:librados-devel-17.2.7-521.gc80e7168.el9.x86_64. 277 kB/s | 126 kB 00:00 2024-02-19T18:47:28.621 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:28.621 INFO:teuthology.orchestra.run.smithi077.stdout:Total 277 kB/s | 126 kB 00:00 2024-02-19T18:47:28.622 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:28.627 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:28.627 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:28.636 INFO:teuthology.orchestra.run.smithi104.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 196 kB/s | 24 kB 00:00 2024-02-19T18:47:28.661 INFO:teuthology.orchestra.run.smithi104.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 121 kB/s | 18 kB 00:00 2024-02-19T18:47:28.681 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:28.682 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:28.695 INFO:teuthology.orchestra.run.smithi104.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2024-02-19T18:47:28.720 INFO:teuthology.orchestra.run.smithi104.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-02-19T18:47:28.753 INFO:teuthology.orchestra.run.smithi104.stdout:(5/13): ceph-mgr-rook-17.2.7-521.gc80e7168.el9. 196 kB/s | 47 kB 00:00 2024-02-19T18:47:28.779 INFO:teuthology.orchestra.run.smithi104.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 729 kB/s | 43 kB 00:00 2024-02-19T18:47:28.796 INFO:teuthology.orchestra.run.smithi104.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.0 MB/s | 211 kB 00:00 2024-02-19T18:47:28.831 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:28.919 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:29.121 INFO:teuthology.orchestra.run.smithi104.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 44 kB/s | 15 kB 00:00 2024-02-19T18:47:29.263 INFO:teuthology.orchestra.run.smithi104.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 63 kB/s | 32 kB 00:00 2024-02-19T18:47:29.360 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:29.446 INFO:teuthology.orchestra.run.smithi104.stdout:(10/13): python3-google-auth-2.27.0-1.el9.noarc 312 kB/s | 203 kB 00:00 2024-02-19T18:47:29.472 INFO:teuthology.orchestra.run.smithi104.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 283 kB/s | 59 kB 00:00 2024-02-19T18:47:29.572 INFO:teuthology.orchestra.run.smithi104.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 716 kB/s | 90 kB 00:00 2024-02-19T18:47:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:29.665 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:29.665 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:29.665 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:29.707 INFO:teuthology.orchestra.run.smithi104.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.7 MB/s | 1.0 MB 00:00 2024-02-19T18:47:29.710 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:29.710 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-02-19T18:47:29.710 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:29.724 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:29.724 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:29.821 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:29.821 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:29.913 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs2 2024-02-19T18:47:30.337 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:30.502 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-02-19T18:47:30.502 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:35 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:30.556 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-02-19T18:47:30.621 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-02-19T18:47:30.621 INFO:teuthology.orchestra.run.smithi077.stdout:Package libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:30.658 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-02-19T18:47:30.675 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:30.676 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:47:30.676 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:30.699 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-02-19T18:47:30.740 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs-devel 2024-02-19T18:47:30.793 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-02-19T18:47:30.901 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 7/13 2024-02-19T18:47:30.958 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-02-19T18:47:31.322 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:36 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:31.489 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 25 k 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 110 k 2024-02-19T18:47:31.490 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:31.657 INFO:teuthology.orchestra.run.smithi077.stdout:libcephfs-devel-17.2.7-521.gc80e7168.el9.x86_64 150 kB/s | 25 kB 00:00 2024-02-19T18:47:31.658 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:31.658 INFO:teuthology.orchestra.run.smithi077.stdout:Total 148 kB/s | 25 kB 00:00 2024-02-19T18:47:31.658 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:31.661 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:31.661 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:31.663 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:31.663 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:31.675 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-02-19T18:47:31.722 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-02-19T18:47:31.752 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:31.756 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-02-19T18:47:31.802 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-02-19T18:47:31.826 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-19T18:47:31.847 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:32.218 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:32.485 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:32.486 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:32.486 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:32.486 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:32.486 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:32.486 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:32.589 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 13/13 2024-02-19T18:47:32.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 1/13 2024-02-19T18:47:32.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-02-19T18:47:32.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-02-19T18:47:32.589 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-02-19T18:47:32.590 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-02-19T18:47:32.590 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-02-19T18:47:32.590 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-02-19T18:47:32.590 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-02-19T18:47:32.590 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-02-19T18:47:32.590 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 10/13 2024-02-19T18:47:32.590 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-02-19T18:47:32.590 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-02-19T18:47:32.649 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados2 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-rook-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-19T18:47:32.964 INFO:teuthology.orchestra.run.smithi104.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-19T18:47:32.965 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-19T18:47:32.965 INFO:teuthology.orchestra.run.smithi104.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-19T18:47:32.965 INFO:teuthology.orchestra.run.smithi104.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-19T18:47:32.965 INFO:teuthology.orchestra.run.smithi104.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-19T18:47:32.965 INFO:teuthology.orchestra.run.smithi104.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-19T18:47:32.965 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:32.965 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:33.220 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-mgr-cephadm 2024-02-19T18:47:33.230 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:38 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:33.347 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:33.396 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:33.397 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:47:33.397 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:33.467 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librbd1 2024-02-19T18:47:33.801 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:51 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:33.967 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:33.968 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-02-19T18:47:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:47:33.968 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-02-19T18:47:33.968 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm noarch 2:17.2.7-521.gc80e7168.el9 ceph-noarch 121 k 2024-02-19T18:47:33.968 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:47:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-02-19T18:47:33.968 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-19T18:47:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-19T18:47:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-19T18:47:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-19T18:47:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:33.969 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:33.969 INFO:teuthology.orchestra.run.smithi104.stdout:======================================================================================= 2024-02-19T18:47:33.969 INFO:teuthology.orchestra.run.smithi104.stdout:Install 6 Packages 2024-02-19T18:47:33.969 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:33.970 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 6.8 M 2024-02-19T18:47:33.970 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 33 M 2024-02-19T18:47:33.970 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:34.047 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:39 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:34.163 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:34.172 INFO:teuthology.orchestra.run.smithi104.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 227 kB 00:00 2024-02-19T18:47:34.213 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:34.214 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:47:34.215 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:34.248 INFO:teuthology.orchestra.run.smithi104.stdout:(2/6): ceph-mgr-cephadm-17.2.7-521.gc80e7168.el 518 kB/s | 121 kB 00:00 2024-02-19T18:47:34.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rados 2024-02-19T18:47:34.356 INFO:teuthology.orchestra.run.smithi104.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-02-19T18:47:34.740 INFO:teuthology.orchestra.run.smithi104.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 124 kB/s | 58 kB 00:00 2024-02-19T18:47:34.798 INFO:teuthology.orchestra.run.smithi104.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 122 kB/s | 51 kB 00:00 2024-02-19T18:47:34.854 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:39 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:34.857 INFO:teuthology.orchestra.run.smithi104.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 801 kB/s | 548 kB 00:00 2024-02-19T18:47:34.861 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:34.861 INFO:teuthology.orchestra.run.smithi104.stdout:Total 7.6 MB/s | 6.8 MB 00:00 2024-02-19T18:47:34.861 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:34.870 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:34.871 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:34.959 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:34.959 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:34.969 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:35.022 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:35.023 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:47:35.023 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:35.092 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rgw 2024-02-19T18:47:35.262 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:35.437 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-02-19T18:47:35.492 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-02-19T18:47:35.669 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:40 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:35.783 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:35.835 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:35.836 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:47:35.836 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:35.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-cephfs 2024-02-19T18:47:36.002 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-02-19T18:47:36.085 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-02-19T18:47:36.141 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-02-19T18:47:36.170 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-19T18:47:36.474 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:41 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:36.587 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:36.638 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:36.639 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:47:36.639 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:36.705 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rbd 2024-02-19T18:47:36.968 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 6/6 2024-02-19T18:47:36.968 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 1/6 2024-02-19T18:47:36.968 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-02-19T18:47:36.968 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-02-19T18:47:36.969 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-02-19T18:47:36.969 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-02-19T18:47:37.268 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-02-19T18:47:37.268 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:37.268 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:37.269 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-mgr-cephadm-2:17.2.7-521.gc80e7168.el9.noarch 2024-02-19T18:47:37.269 INFO:teuthology.orchestra.run.smithi104.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-02-19T18:47:37.269 INFO:teuthology.orchestra.run.smithi104.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-19T18:47:37.269 INFO:teuthology.orchestra.run.smithi104.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-19T18:47:37.269 INFO:teuthology.orchestra.run.smithi104.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-19T18:47:37.269 INFO:teuthology.orchestra.run.smithi104.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-19T18:47:37.269 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:37.269 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:37.283 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:42 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:37.398 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:37.451 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:37.452 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2024-02-19T18:47:37.452 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:37.511 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install ceph-fuse 2024-02-19T18:47:37.521 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-fuse 2024-02-19T18:47:38.089 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:56 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:38.097 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:43 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:38.256 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:38.257 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:38.257 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repository Size 2024-02-19T18:47:38.257 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:38.257 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:38.257 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 812 k 2024-02-19T18:47:38.257 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout:Install 2 Packages 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 891 k 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 2.6 M 2024-02-19T18:47:38.258 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:38.263 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:38.264 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:38.264 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-19T18:47:38.264 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:38.264 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:38.264 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 81 k 2024-02-19T18:47:38.265 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:38.265 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:38.265 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:38.265 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-19T18:47:38.265 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:38.265 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 81 k 2024-02-19T18:47:38.265 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 222 k 2024-02-19T18:47:38.265 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:38.375 INFO:teuthology.orchestra.run.smithi104.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 676 kB/s | 79 kB 00:00 2024-02-19T18:47:38.477 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-fuse-17.2.7-521.gc80e7168.el9.x86_64.rpm 384 kB/s | 81 kB 00:00 2024-02-19T18:47:38.478 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:38.478 INFO:teuthology.orchestra.run.smithi077.stdout:Total 382 kB/s | 81 kB 00:00 2024-02-19T18:47:38.478 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:38.485 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:38.485 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:38.539 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:38.539 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:38.674 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:38.783 INFO:teuthology.orchestra.run.smithi104.stdout:(2/2): ceph-fuse-17.2.7-521.gc80e7168.el9.x86_6 1.5 MB/s | 812 kB 00:00 2024-02-19T18:47:38.784 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:38.784 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:38.785 INFO:teuthology.orchestra.run.smithi104.stdout:Total 1.6 MB/s | 891 kB 00:00 2024-02-19T18:47:38.785 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:38.805 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:38.806 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:38.880 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:38.880 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:39.061 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:39.172 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:39.187 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-02-19T18:47:39.265 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-19T18:47:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:39.440 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:39.440 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:39.441 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:39.441 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:39.682 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-mirror 2024-02-19T18:47:39.963 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2/2 2024-02-19T18:47:39.964 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/2 2024-02-19T18:47:40.271 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:45 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:40.281 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-02-19T18:47:40.281 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:40.281 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:40.281 INFO:teuthology.orchestra.run.smithi104.stdout: ceph-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-02-19T18:47:40.281 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:40.281 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:40.442 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:40.442 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 2.9 M 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 12 M 2024-02-19T18:47:40.443 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:40.546 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados-devel 2024-02-19T18:47:41.089 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-mirror-17.2.7-521.gc80e7168.el9.x86_64.rpm 4.6 MB/s | 2.9 MB 00:00 2024-02-19T18:47:41.090 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:41.090 INFO:teuthology.orchestra.run.smithi077.stdout:Total 4.5 MB/s | 2.9 MB 00:00 2024-02-19T18:47:41.090 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:41.099 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:41.099 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:41.142 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:01:59 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:41.172 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:41.173 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:41.306 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 126 k 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:41.307 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-19T18:47:41.308 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:41.308 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 126 k 2024-02-19T18:47:41.308 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 448 k 2024-02-19T18:47:41.308 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:41.584 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:41.631 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:41.655 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:41.655 INFO:teuthology.orchestra.run.smithi077.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:47:41.655 INFO:teuthology.orchestra.run.smithi077.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-19T18:47:41.655 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:47:41.656 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:47:41.656 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:41.670 INFO:teuthology.orchestra.run.smithi104.stdout:librados-devel-17.2.7-521.gc80e7168.el9.x86_64. 349 kB/s | 126 kB 00:00 2024-02-19T18:47:41.671 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:41.671 INFO:teuthology.orchestra.run.smithi104.stdout:Total 347 kB/s | 126 kB 00:00 2024-02-19T18:47:41.671 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:41.677 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:41.677 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:41.731 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:41.731 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:41.872 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:41.968 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:42.423 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:42.734 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:42.734 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:42.734 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:42.734 INFO:teuthology.orchestra.run.smithi104.stdout: librados-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:42.734 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:42.734 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:42.847 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:42.848 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:42.848 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:42.848 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:42.848 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:42.848 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:43.003 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs2 2024-02-19T18:47:43.104 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-nbd 2024-02-19T18:47:43.595 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:01 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:43.698 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:48 ago on Mon 19 Feb 2024 06:45:55 PM UTC. 2024-02-19T18:47:43.712 INFO:teuthology.orchestra.run.smithi104.stdout:Package libcephfs2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:43.764 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:43.765 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:47:43.765 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:43.837 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install libcephfs-devel 2024-02-19T18:47:43.865 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2024-02-19T18:47:43.866 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:43.866 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2024-02-19T18:47:43.866 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 165 k 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 165 k 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 477 k 2024-02-19T18:47:43.867 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2024-02-19T18:47:44.092 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-nbd-17.2.7-521.gc80e7168.el9.x86_64.rpm 732 kB/s | 165 kB 00:00 2024-02-19T18:47:44.093 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:44.093 INFO:teuthology.orchestra.run.smithi077.stdout:Total 727 kB/s | 165 kB 00:00 2024-02-19T18:47:44.093 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2024-02-19T18:47:44.101 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2024-02-19T18:47:44.101 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2024-02-19T18:47:44.157 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2024-02-19T18:47:44.157 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2024-02-19T18:47:44.301 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2024-02-19T18:47:44.408 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:44.417 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:02 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:44.580 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:44.581 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:44.581 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-02-19T18:47:44.581 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:44.581 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:44.581 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel x86_64 2:17.2.7-521.gc80e7168.el9 ceph 25 k 2024-02-19T18:47:44.581 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:44.581 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:44.582 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:44.582 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-19T18:47:44.582 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:44.582 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 25 k 2024-02-19T18:47:44.582 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 110 k 2024-02-19T18:47:44.582 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:44.749 INFO:teuthology.orchestra.run.smithi104.stdout:libcephfs-devel-17.2.7-521.gc80e7168.el9.x86_64 150 kB/s | 25 kB 00:00 2024-02-19T18:47:44.749 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:44.749 INFO:teuthology.orchestra.run.smithi104.stdout:Total 149 kB/s | 25 kB 00:00 2024-02-19T18:47:44.749 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:44.752 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:44.753 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:44.754 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:44.754 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:44.835 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:44.896 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:44.972 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:45.184 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:45.184 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:45.184 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2024-02-19T18:47:45.185 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:45.185 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:47:45.185 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2024-02-19T18:47:45.378 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:45.439 DEBUG:teuthology.parallel:result is None 2024-02-19T18:47:45.640 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:45.640 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:45.641 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:45.641 INFO:teuthology.orchestra.run.smithi104.stdout: libcephfs-devel-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:45.641 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:45.641 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:45.831 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librados2 2024-02-19T18:47:46.409 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:04 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:46.523 INFO:teuthology.orchestra.run.smithi104.stdout:Package librados2-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:46.574 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:46.575 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:47:46.575 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:46.644 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install librbd1 2024-02-19T18:47:47.222 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:05 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:47.336 INFO:teuthology.orchestra.run.smithi104.stdout:Package librbd1-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:47.387 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:47.388 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:47:47.388 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:47.464 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rados 2024-02-19T18:47:48.042 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:06 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:48.155 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rados-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:48.207 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:48.208 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:47:48.208 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:48.285 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rgw 2024-02-19T18:47:48.867 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:06 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:48.983 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rgw-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:49.035 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:49.037 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:47:49.037 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:49.106 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-cephfs 2024-02-19T18:47:49.682 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:07 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:49.796 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-cephfs-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:49.848 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:49.849 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:47:49.849 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:49.918 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install python3-rbd 2024-02-19T18:47:50.495 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:08 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:50.611 INFO:teuthology.orchestra.run.smithi104.stdout:Package python3-rbd-2:17.2.7-521.gc80e7168.el9.x86_64 is already installed. 2024-02-19T18:47:50.664 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:50.665 INFO:teuthology.orchestra.run.smithi104.stdout:Nothing to do. 2024-02-19T18:47:50.665 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:50.738 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-fuse 2024-02-19T18:47:51.319 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:09 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:51.484 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:51.484 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse x86_64 2:17.2.7-521.gc80e7168.el9 ceph 81 k 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 81 k 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 222 k 2024-02-19T18:47:51.485 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:51.743 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-fuse-17.2.7-521.gc80e7168.el9.x86_64.rpm 317 kB/s | 81 kB 00:00 2024-02-19T18:47:51.743 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:51.743 INFO:teuthology.orchestra.run.smithi104.stdout:Total 315 kB/s | 81 kB 00:00 2024-02-19T18:47:51.744 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:51.750 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:51.750 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:51.805 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:51.805 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:51.939 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:52.031 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:52.477 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:52.798 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:52.798 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:52.798 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:52.798 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-fuse-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:52.798 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:52.798 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:53.052 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-mirror 2024-02-19T18:47:53.648 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:11 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:53.815 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:53.821 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:53.821 INFO:teuthology.orchestra.run.smithi104.stdout: Package Arch Version Repo Size 2024-02-19T18:47:53.821 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:53.821 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:53.821 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror x86_64 2:17.2.7-521.gc80e7168.el9 ceph 2.9 M 2024-02-19T18:47:53.821 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:53.821 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:53.822 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:53.822 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-19T18:47:53.822 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:53.822 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 2.9 M 2024-02-19T18:47:53.822 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 12 M 2024-02-19T18:47:53.822 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:54.286 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-mirror-17.2.7-521.gc80e7168.el9.x86_64.rpm 6.3 MB/s | 2.9 MB 00:00 2024-02-19T18:47:54.287 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:54.287 INFO:teuthology.orchestra.run.smithi104.stdout:Total 6.2 MB/s | 2.9 MB 00:00 2024-02-19T18:47:54.287 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:54.297 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:54.297 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:54.367 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:54.367 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:54.781 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:54.845 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:54.868 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:54.868 INFO:teuthology.orchestra.run.smithi104.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-19T18:47:54.868 INFO:teuthology.orchestra.run.smithi104.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-19T18:47:54.869 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:47:54.869 INFO:teuthology.orchestra.run.smithi104.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-19T18:47:54.869 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:55.967 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:55.967 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:55.967 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:55.967 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-mirror-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:55.967 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:55.967 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:56.184 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum -y install rbd-nbd 2024-02-19T18:47:56.778 INFO:teuthology.orchestra.run.smithi104.stdout:Last metadata expiration check: 0:02:14 ago on Mon 19 Feb 2024 06:45:42 PM UTC. 2024-02-19T18:47:56.943 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-02-19T18:47:56.944 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:56.944 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-02-19T18:47:56.944 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:56.944 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-02-19T18:47:56.944 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd x86_64 2:17.2.7-521.gc80e7168.el9 ceph 165 k 2024-02-19T18:47:56.944 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:56.945 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-02-19T18:47:56.945 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-02-19T18:47:56.945 INFO:teuthology.orchestra.run.smithi104.stdout:Install 1 Package 2024-02-19T18:47:56.945 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:56.946 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 165 k 2024-02-19T18:47:56.946 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 477 k 2024-02-19T18:47:56.946 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-02-19T18:47:57.164 INFO:teuthology.orchestra.run.smithi104.stdout:rbd-nbd-17.2.7-521.gc80e7168.el9.x86_64.rpm 751 kB/s | 165 kB 00:00 2024-02-19T18:47:57.165 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:47:57.165 INFO:teuthology.orchestra.run.smithi104.stdout:Total 747 kB/s | 165 kB 00:00 2024-02-19T18:47:57.165 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-02-19T18:47:57.172 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-02-19T18:47:57.172 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-02-19T18:47:57.228 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-02-19T18:47:57.228 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-02-19T18:47:57.380 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-02-19T18:47:57.486 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:57.941 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:58.289 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 1/1 2024-02-19T18:47:58.289 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:58.290 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-02-19T18:47:58.291 INFO:teuthology.orchestra.run.smithi104.stdout: rbd-nbd-2:17.2.7-521.gc80e7168.el9.x86_64 2024-02-19T18:47:58.291 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:47:58.291 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-02-19T18:47:58.563 DEBUG:teuthology.parallel:result is None 2024-02-19T18:47:58.564 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:47:58.565 INFO:teuthology.packaging:ref: None 2024-02-19T18:47:58.565 INFO:teuthology.packaging:tag: None 2024-02-19T18:47:58.565 INFO:teuthology.packaging:branch: quincy 2024-02-19T18:47:58.565 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:47:58.566 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-19T18:47:58.933 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T18:47:58.958 INFO:teuthology.orchestra.run.smithi077.stdout:17.2.7-521.gc80e7168.el9 2024-02-19T18:47:58.959 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-521.gc80e7168.el9 2024-02-19T18:47:58.959 INFO:teuthology.task.install:The correct ceph version 17.2.7-521.gc80e7168 is installed. 2024-02-19T18:47:58.961 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-19T18:47:58.961 INFO:teuthology.packaging:ref: None 2024-02-19T18:47:58.961 INFO:teuthology.packaging:tag: None 2024-02-19T18:47:58.961 INFO:teuthology.packaging:branch: quincy 2024-02-19T18:47:58.961 INFO:teuthology.packaging:sha1: f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:47:58.961 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-02-19T18:47:59.186 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-19T18:47:59.207 INFO:teuthology.orchestra.run.smithi104.stdout:17.2.7-521.gc80e7168.el9 2024-02-19T18:47:59.208 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-521.gc80e7168.el9 2024-02-19T18:47:59.208 INFO:teuthology.task.install:The correct ceph version 17.2.7-521.gc80e7168 is installed. 2024-02-19T18:47:59.210 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-19T18:47:59.210 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:47:59.210 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T18:47:59.242 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:47:59.242 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-19T18:47:59.276 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-19T18:47:59.276 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:47:59.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T18:47:59.311 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T18:47:59.380 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:47:59.380 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/daemon-helper 2024-02-19T18:47:59.407 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-19T18:47:59.475 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-19T18:47:59.475 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:47:59.475 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T18:47:59.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T18:47:59.572 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:47:59.572 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-19T18:47:59.598 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-19T18:47:59.666 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-19T18:47:59.666 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:47:59.666 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T18:47:59.694 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T18:47:59.761 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:47:59.761 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/usr/bin/stdin-killer 2024-02-19T18:47:59.789 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-19T18:47:59.856 INFO:teuthology.run_tasks:Running task print... 2024-02-19T18:47:59.866 INFO:teuthology.task.print:**** done install task... 2024-02-19T18:47:59.866 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-19T18:48:00.012 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'f78a58c0ffd401d1493058a1022c35f011d65275'} 2024-02-19T18:48:00.012 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-02-19T18:48:00.013 INFO:tasks.cephadm:Cluster fsid is 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:48:00.013 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-19T18:48:00.013 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-02-19T18:48:00.013 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi077': '172.21.15.77', 'mon.smithi104': '172.21.15.104'} 2024-02-19T18:48:00.013 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-19T18:48:00.013 DEBUG:teuthology.orchestra.run.smithi077:> sudo hostname $(hostname -s) 2024-02-19T18:48:00.041 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-02-19T18:48:00.069 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-02-19T18:48:00.070 DEBUG:teuthology.orchestra.run.smithi077:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T18:48:00.703 INFO:teuthology.orchestra.run.smithi077.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 19 18:48 /home/ubuntu/cephtest/cephadm 2024-02-19T18:48:00.704 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-19T18:48:00.889 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 367151 Feb 19 18:48 /home/ubuntu/cephtest/cephadm 2024-02-19T18:48:00.889 DEBUG:teuthology.orchestra.run.smithi077:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T18:48:00.909 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-19T18:48:00.936 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-02-19T18:48:00.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-19T18:48:00.952 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-02-19T18:48:01.778 INFO:teuthology.orchestra.run.smithi077.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-19T18:48:01.842 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-19T18:49:30.754 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-02-19T18:49:30.754 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)", 2024-02-19T18:49:30.755 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e", 2024-02-19T18:49:30.755 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-02-19T18:49:30.755 INFO:teuthology.orchestra.run.smithi104.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b" 2024-02-19T18:49:30.755 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-02-19T18:49:30.755 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-02-19T18:49:30.918 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T18:49:30.918 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph_version": "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)", 2024-02-19T18:49:30.918 INFO:teuthology.orchestra.run.smithi077.stdout: "image_id": "10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e", 2024-02-19T18:49:30.918 INFO:teuthology.orchestra.run.smithi077.stdout: "repo_digests": [ 2024-02-19T18:49:30.918 INFO:teuthology.orchestra.run.smithi077.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b" 2024-02-19T18:49:30.918 INFO:teuthology.orchestra.run.smithi077.stdout: ] 2024-02-19T18:49:30.918 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T18:49:30.937 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph 2024-02-19T18:49:30.992 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-02-19T18:49:31.020 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /etc/ceph 2024-02-19T18:49:31.047 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-02-19T18:49:31.087 INFO:tasks.cephadm:Writing seed config... 2024-02-19T18:49:31.088 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-02-19T18:49:31.088 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-02-19T18:49:31.088 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-19T18:49:31.088 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-19T18:49:31.088 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-19T18:49:31.088 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-19T18:49:31.088 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-19T18:49:31.089 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-19T18:49:31.090 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-19T18:49:31.091 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-19T18:49:31.091 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:49:31.091 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-19T18:49:31.106 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 6778a95a-cf57-11ee-95bb-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-19T18:49:31.107 DEBUG:teuthology.orchestra.run.smithi077:mon.smithi077> sudo journalctl -f -n 0 -u ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi077.service 2024-02-19T18:49:31.149 INFO:tasks.cephadm:Bootstrapping... 2024-02-19T18:49:31.149 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.77 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:49:31.352 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2024-02-19T18:49:31.352 INFO:teuthology.orchestra.run.smithi077.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '6778a95a-cf57-11ee-95bb-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.77', '--skip-admin-label'] 2024-02-19T18:49:31.376 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:49:31.376 INFO:teuthology.orchestra.run.smithi077.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-19T18:49:31.376 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying podman|docker is present... 2024-02-19T18:49:31.400 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:49:31.400 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying lvm2 is present... 2024-02-19T18:49:31.400 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying time synchronization is in place... 2024-02-19T18:49:31.408 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-19T18:49:31.408 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T18:49:31.414 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-19T18:49:31.414 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-02-19T18:49:31.421 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout enabled 2024-02-19T18:49:31.427 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout active 2024-02-19T18:49:31.427 INFO:teuthology.orchestra.run.smithi077.stdout:Unit chronyd.service is enabled and running 2024-02-19T18:49:31.427 INFO:teuthology.orchestra.run.smithi077.stdout:Repeating the final host check... 2024-02-19T18:49:31.450 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-19T18:49:31.450 INFO:teuthology.orchestra.run.smithi077.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-19T18:49:31.451 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl is present 2024-02-19T18:49:31.451 INFO:teuthology.orchestra.run.smithi077.stdout:lvcreate is present 2024-02-19T18:49:31.457 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-19T18:49:31.457 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-19T18:49:31.463 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-19T18:49:31.463 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-02-19T18:49:31.469 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout enabled 2024-02-19T18:49:31.476 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout active 2024-02-19T18:49:31.476 INFO:teuthology.orchestra.run.smithi077.stdout:Unit chronyd.service is enabled and running 2024-02-19T18:49:31.476 INFO:teuthology.orchestra.run.smithi077.stdout:Host looks OK 2024-02-19T18:49:31.476 INFO:teuthology.orchestra.run.smithi077.stdout:Cluster fsid: 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:49:31.476 INFO:teuthology.orchestra.run.smithi077.stdout:Acquiring lock 140255382055424 on /run/cephadm/6778a95a-cf57-11ee-95bb-87774f69a715.lock 2024-02-19T18:49:31.476 INFO:teuthology.orchestra.run.smithi077.stdout:Lock 140255382055424 acquired on /run/cephadm/6778a95a-cf57-11ee-95bb-87774f69a715.lock 2024-02-19T18:49:31.476 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying IP 172.21.15.77 port 3300 ... 2024-02-19T18:49:31.477 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying IP 172.21.15.77 port 6789 ... 2024-02-19T18:49:31.477 INFO:teuthology.orchestra.run.smithi077.stdout:Base mon IP(s) is [172.21.15.77:3300, 172.21.15.77:6789], mon addrv is [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-02-19T18:49:31.480 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.77 metric 100 2024-02-19T18:49:31.480 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.77 metric 100 2024-02-19T18:49:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-19T18:49:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-02-19T18:49:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1768sec hoplimit 64 pref medium 2024-02-19T18:49:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-19T18:49:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-19T18:49:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-19T18:49:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-02-19T18:49:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-02-19T18:49:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-19T18:49:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-02-19T18:49:31.487 INFO:teuthology.orchestra.run.smithi077.stdout:Mon IP `172.21.15.77` is in CIDR network `172.21.0.0/20` 2024-02-19T18:49:31.487 INFO:teuthology.orchestra.run.smithi077.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-19T18:49:31.487 INFO:teuthology.orchestra.run.smithi077.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-19T18:49:31.488 INFO:teuthology.orchestra.run.smithi077.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-19T18:49:32.002 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stdout 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e 2024-02-19T18:49:32.002 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-02-19T18:49:32.002 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Getting image source signatures 2024-02-19T18:49:32.002 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying blob sha256:f8275265412e57f06c462f3ab36d3ba9cb1e55305ee9294fe928d41e45938854 2024-02-19T18:49:32.002 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying blob sha256:0f642e61a11afb42c0ccb5b05f5809250ad643ed143780d6ff49a5a4c492fc38 2024-02-19T18:49:32.002 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Copying config sha256:10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e 2024-02-19T18:49:32.002 INFO:teuthology.orchestra.run.smithi077.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-19T18:49:32.822 INFO:teuthology.orchestra.run.smithi077.stdout:ceph: stdout ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable) 2024-02-19T18:49:32.823 INFO:teuthology.orchestra.run.smithi077.stdout:Ceph version: ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable) 2024-02-19T18:49:32.823 INFO:teuthology.orchestra.run.smithi077.stdout:Extracting ceph user uid/gid from container image... 2024-02-19T18:49:33.662 INFO:teuthology.orchestra.run.smithi077.stdout:stat: stdout 167 167 2024-02-19T18:49:33.662 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial keys... 2024-02-19T18:49:34.592 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQC9otNl+1TsORAA5pEW8SqOHIpnOHLECqkMMg== 2024-02-19T18:49:35.465 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQC+otNlcsS7NhAAl5t1kh9o7Jq9qxe/DXgTYQ== 2024-02-19T18:49:36.395 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph-authtool: stdout AQC/otNlFvGWLRAAfdpoeqQL2o2TCxeDeyEPUg== 2024-02-19T18:49:36.397 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial monmap... 2024-02-19T18:49:37.294 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T18:49:37.294 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-02-19T18:49:37.294 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:49:37.294 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T18:49:37.294 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool for smithi077 [v2:172.21.15.77:3300,v1:172.21.15.77:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-19T18:49:37.294 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = octopus 2024-02-19T18:49:37.294 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: set fsid to 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:49:37.294 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-19T18:49:37.295 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:49:37.295 INFO:teuthology.orchestra.run.smithi077.stdout:Creating mon... 2024-02-19T18:49:38.240 INFO:teuthology.orchestra.run.smithi077.stdout:create mon.smithi077 on 2024-02-19T18:49:38.443 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-19T18:49:38.594 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-19T18:49:38.746 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-6778a95a-cf57-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-6778a95a-cf57-11ee-95bb-87774f69a715.target. 2024-02-19T18:49:38.747 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-6778a95a-cf57-11ee-95bb-87774f69a715.target → /etc/systemd/system/ceph-6778a95a-cf57-11ee-95bb-87774f69a715.target. 2024-02-19T18:49:38.945 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi077 2024-02-19T18:49:38.945 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi077.service: Unit ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi077.service not loaded. 2024-02-19T18:49:39.116 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6778a95a-cf57-11ee-95bb-87774f69a715.target.wants/ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi077.service → /etc/systemd/system/ceph-6778a95a-cf57-11ee-95bb-87774f69a715@.service. 2024-02-19T18:49:40.533 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:49:40.534 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-02-19T18:49:40.542 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:49:40.542 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-02-19T18:49:40.542 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-02-19T18:49:40.542 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to enable service . firewalld.service is not available 2024-02-19T18:49:40.543 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mon to start... 2024-02-19T18:49:40.543 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mon... 2024-02-19T18:49:40.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:40 smithi077 ceph-mon[28877]: mgrmap e1: no daemons active 2024-02-19T18:49:43.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:43 smithi077 ceph-mon[28877]: from='client.? 172.21.15.77:0/3596224963' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout cluster: 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout id: 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout services: 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi077 (age 3s) 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-19T18:49:49.222 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout data: 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout pgs: 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.486+0000 7f6a54918700 1 Processor -- start 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.486+0000 7f6a54918700 1 -- start start 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.486+0000 7f6a54918700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50104230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.486+0000 7f6a54918700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a50104770 con 0x7f6a50103e20 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.486+0000 7f6a4e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50104230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.487+0000 7f6a4e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50104230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59552/0 (socket says 172.21.15.77:59552) 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.487+0000 7f6a4e59c700 1 -- 172.21.15.77:0/1354971221 learned_addr learned my addr 172.21.15.77:0/1354971221 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.487+0000 7f6a4e59c700 1 -- 172.21.15.77:0/1354971221 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a501048b0 con 0x7f6a50103e20 2024-02-19T18:49:49.223 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.487+0000 7f6a4e59c700 1 --2- 172.21.15.77:0/1354971221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50104230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6a400098e0 tx=0x7f6a40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e309350056cc0d04 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.487+0000 7f6a4d59a700 1 -- 172.21.15.77:0/1354971221 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a4000b860 con 0x7f6a50103e20 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.487+0000 7f6a4d59a700 1 -- 172.21.15.77:0/1354971221 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4000b9c0 con 0x7f6a50103e20 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.487+0000 7f6a4d59a700 1 -- 172.21.15.77:0/1354971221 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a4000bc90 con 0x7f6a50103e20 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 -- 172.21.15.77:0/1354971221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 msgr2=0x7f6a50104230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 --2- 172.21.15.77:0/1354971221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50104230 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f6a400098e0 tx=0x7f6a40009250 comp rx=0 tx=0).stop 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 -- 172.21.15.77:0/1354971221 shutdown_connections 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 --2- 172.21.15.77:0/1354971221 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50104230 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 -- 172.21.15.77:0/1354971221 >> 172.21.15.77:0/1354971221 conn(0x7f6a500fc580 msgr2=0x7f6a500fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 -- 172.21.15.77:0/1354971221 shutdown_connections 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 -- 172.21.15.77:0/1354971221 wait complete. 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 Processor -- start 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 -- start start 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50103660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.488+0000 7f6a54918700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a50101ae0 con 0x7f6a50103e20 2024-02-19T18:49:49.224 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a4e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50103660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a4e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50103660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:59566/0 (socket says 172.21.15.77:59566) 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a4e59c700 1 -- 172.21.15.77:0/3596224963 learned_addr learned my addr 172.21.15.77:0/3596224963 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a4e59c700 1 -- 172.21.15.77:0/3596224963 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a400094f0 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a4e59c700 1 --2- 172.21.15.77:0/3596224963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50103660 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a40003080 tx=0x7f6a40016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a3effd700 1 -- 172.21.15.77:0/3596224963 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a4001c5c0 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a3effd700 1 -- 172.21.15.77:0/3596224963 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a4001c720 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a3effd700 1 -- 172.21.15.77:0/3596224963 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a4001ca30 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a50101ce0 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a50102170 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a3effd700 1 -- 172.21.15.77:0/3596224963 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6a4001a450 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.489+0000 7f6a3effd700 1 -- 172.21.15.77:0/3596224963 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6a4002e790 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.490+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a5010acf0 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.492+0000 7f6a3effd700 1 -- 172.21.15.77:0/3596224963 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6a4001a890 con 0x7f6a50103e20 2024-02-19T18:49:49.225 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.563+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f6a501008b0 con 0x7f6a50103e20 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.564+0000 7f6a3effd700 1 -- 172.21.15.77:0/3596224963 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+318 (secure 0 0 0) 0x7f6a4001aa30 con 0x7f6a50103e20 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.564+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 msgr2=0x7f6a50103660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.564+0000 7f6a54918700 1 --2- 172.21.15.77:0/3596224963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50103660 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f6a40003080 tx=0x7f6a40016000 comp rx=0 tx=0).stop 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.564+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 shutdown_connections 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.564+0000 7f6a54918700 1 --2- 172.21.15.77:0/3596224963 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6a50103e20 0x7f6a50103660 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.564+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 >> 172.21.15.77:0/3596224963 conn(0x7f6a500fc580 msgr2=0x7f6a50107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.564+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 shutdown_connections 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:43.564+0000 7f6a54918700 1 -- 172.21.15.77:0/3596224963 wait complete. 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:mon is available 2024-02-19T18:49:49.226 INFO:teuthology.orchestra.run.smithi077.stdout:Assimilating anything we can from ceph.conf... 2024-02-19T18:49:50.252 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:49:50.252 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [global] 2024-02-19T18:49:50.252 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout fsid = 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:49:50.252 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.77:3300,v1:172.21.15.77:6789] 2024-02-19T18:49:50.252 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-19T18:49:50.252 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [osd] 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.619+0000 7ffb6f917700 1 Processor -- start 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.619+0000 7ffb6f917700 1 -- start start 2024-02-19T18:49:50.253 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb6f917700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb6f917700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681030c0 con 0x7ffb68102770 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb6d6b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb6d6b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58698/0 (socket says 172.21.15.77:58698) 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb6d6b3700 1 -- 172.21.15.77:0/2551482538 learned_addr learned my addr 172.21.15.77:0/2551482538 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb6d6b3700 1 -- 172.21.15.77:0/2551482538 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb68103200 con 0x7ffb68102770 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb6d6b3700 1 --2- 172.21.15.77:0/2551482538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68102b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ffb580098e0 tx=0x7ffb58009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dcf03454c84454a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb5ffff700 1 -- 172.21.15.77:0/2551482538 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5800b860 con 0x7ffb68102770 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb5ffff700 1 -- 172.21.15.77:0/2551482538 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5800b9c0 con 0x7ffb68102770 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.620+0000 7ffb5ffff700 1 -- 172.21.15.77:0/2551482538 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5800bc90 con 0x7ffb68102770 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.621+0000 7ffb6f917700 1 -- 172.21.15.77:0/2551482538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 msgr2=0x7ffb68102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.621+0000 7ffb6f917700 1 --2- 172.21.15.77:0/2551482538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68102b80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ffb580098e0 tx=0x7ffb58009250 comp rx=0 tx=0).stop 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.621+0000 7ffb6f917700 1 -- 172.21.15.77:0/2551482538 shutdown_connections 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.621+0000 7ffb6f917700 1 --2- 172.21.15.77:0/2551482538 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68102b80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:50.254 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.621+0000 7ffb6f917700 1 -- 172.21.15.77:0/2551482538 >> 172.21.15.77:0/2551482538 conn(0x7ffb680fded0 msgr2=0x7ffb68100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.621+0000 7ffb6f917700 1 -- 172.21.15.77:0/2551482538 shutdown_connections 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.621+0000 7ffb6f917700 1 -- 172.21.15.77:0/2551482538 wait complete. 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6f917700 1 Processor -- start 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6f917700 1 -- start start 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6f917700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68110dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6f917700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68111310 con 0x7ffb68102770 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6d6b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68110dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6d6b3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68110dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58710/0 (socket says 172.21.15.77:58710) 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6d6b3700 1 -- 172.21.15.77:0/1747004108 learned_addr learned my addr 172.21.15.77:0/1747004108 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6d6b3700 1 -- 172.21.15.77:0/1747004108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb580094f0 con 0x7ffb68102770 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6d6b3700 1 --2- 172.21.15.77:0/1747004108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68110dd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ffb58003080 tx=0x7ffb58016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb5dffb700 1 -- 172.21.15.77:0/1747004108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5801c5c0 con 0x7ffb68102770 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb5dffb700 1 -- 172.21.15.77:0/1747004108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb5801c720 con 0x7ffb68102770 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.622+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb68111510 con 0x7ffb68102770 2024-02-19T18:49:50.255 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.623+0000 7ffb5dffb700 1 -- 172.21.15.77:0/1747004108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5801ca30 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.623+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb68112100 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.623+0000 7ffb5dffb700 1 -- 172.21.15.77:0/1747004108 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ffb5801a450 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.623+0000 7ffb5dffb700 1 -- 172.21.15.77:0/1747004108 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ffb5802e790 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.624+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6805f5c0 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.626+0000 7ffb5dffb700 1 -- 172.21.15.77:0/1747004108 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ffb5801a890 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.692+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ffb68103b90 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.695+0000 7ffb5dffb700 1 -- 172.21.15.77:0/1747004108 <== mon.0 v2:172.21.15.77:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ffb5802ee80 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.695+0000 7ffb5dffb700 1 -- 172.21.15.77:0/1747004108 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7ffb580247d0 con 0x7ffb68102770 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.696+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 msgr2=0x7ffb68110dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.696+0000 7ffb6f917700 1 --2- 172.21.15.77:0/1747004108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68110dd0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7ffb58003080 tx=0x7ffb58016000 comp rx=0 tx=0).stop 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.696+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 shutdown_connections 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.696+0000 7ffb6f917700 1 --2- 172.21.15.77:0/1747004108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffb68102770 0x7ffb68110dd0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.696+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 >> 172.21.15.77:0/1747004108 conn(0x7ffb680fded0 msgr2=0x7ffb681054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:50.256 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.696+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 shutdown_connections 2024-02-19T18:49:50.257 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:49.696+0000 7ffb6f917700 1 -- 172.21.15.77:0/1747004108 wait complete. 2024-02-19T18:49:50.257 INFO:teuthology.orchestra.run.smithi077.stdout:Generating new minimal ceph.conf... 2024-02-19T18:49:50.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:50 smithi077 ceph-mon[28877]: from='client.? 172.21.15.77:0/1747004108' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T18:49:50.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:50 smithi077 ceph-mon[28877]: from='client.? 172.21.15.77:0/1747004108' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.624+0000 7fd3effed700 1 Processor -- start 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.624+0000 7fd3effed700 1 -- start start 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.624+0000 7fd3effed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8104190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.624+0000 7fd3effed700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e81046d0 con 0x7fd3e8103d80 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.624+0000 7fd3edd89700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.624+0000 7fd3edd89700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8104190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58718/0 (socket says 172.21.15.77:58718) 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.624+0000 7fd3edd89700 1 -- 172.21.15.77:0/2372304477 learned_addr learned my addr 172.21.15.77:0/2372304477 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3edd89700 1 -- 172.21.15.77:0/2372304477 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e8104810 con 0x7fd3e8103d80 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3edd89700 1 --2- 172.21.15.77:0/2372304477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8104190 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40098e0 tx=0x7fd3e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341528a461eaa5c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3ecd87700 1 -- 172.21.15.77:0/2372304477 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e400b860 con 0x7fd3e8103d80 2024-02-19T18:49:51.324 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3ecd87700 1 -- 172.21.15.77:0/2372304477 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd3e400be50 con 0x7fd3e8103d80 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3ecd87700 1 -- 172.21.15.77:0/2372304477 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e4010380 con 0x7fd3e8103d80 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3effed700 1 -- 172.21.15.77:0/2372304477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 msgr2=0x7fd3e8104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3effed700 1 --2- 172.21.15.77:0/2372304477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8104190 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd3e40098e0 tx=0x7fd3e4009250 comp rx=0 tx=0).stop 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3effed700 1 -- 172.21.15.77:0/2372304477 shutdown_connections 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3effed700 1 --2- 172.21.15.77:0/2372304477 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8104190 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.625+0000 7fd3effed700 1 -- 172.21.15.77:0/2372304477 >> 172.21.15.77:0/2372304477 conn(0x7fd3e80fc580 msgr2=0x7fd3e80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3effed700 1 -- 172.21.15.77:0/2372304477 shutdown_connections 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3effed700 1 -- 172.21.15.77:0/2372304477 wait complete. 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3effed700 1 Processor -- start 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3effed700 1 -- start start 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3effed700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3effed700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e8111d60 con 0x7fd3e8103d80 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3edd89700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:51.325 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3edd89700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58724/0 (socket says 172.21.15.77:58724) 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.626+0000 7fd3edd89700 1 -- 172.21.15.77:0/2479641094 learned_addr learned my addr 172.21.15.77:0/2479641094 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3edd89700 1 -- 172.21.15.77:0/2479641094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e40094f0 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3edd89700 1 --2- 172.21.15.77:0/2479641094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8111820 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400b440 tx=0x7fd3e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3de7fc700 1 -- 172.21.15.77:0/2479641094 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e40107c0 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3de7fc700 1 -- 172.21.15.77:0/2479641094 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fd3e4010db0 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e8111f60 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3de7fc700 1 -- 172.21.15.77:0/2479641094 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3e401ada0 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e8112c00 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3de7fc700 1 -- 172.21.15.77:0/2479641094 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd3e402e6c0 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.627+0000 7fd3de7fc700 1 -- 172.21.15.77:0/2479641094 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd3e4024950 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.628+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e805f5c0 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.630+0000 7fd3de7fc700 1 -- 172.21.15.77:0/2479641094 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fd3e4024bb0 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.696+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd3e8064ad0 con 0x7fd3e8103d80 2024-02-19T18:49:51.326 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.697+0000 7fd3de7fc700 1 -- 172.21.15.77:0/2479641094 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd3e4024d50 con 0x7fd3e8103d80 2024-02-19T18:49:51.327 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.697+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 msgr2=0x7fd3e8111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:51.327 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.697+0000 7fd3effed700 1 --2- 172.21.15.77:0/2479641094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8111820 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400b440 tx=0x7fd3e4009250 comp rx=0 tx=0).stop 2024-02-19T18:49:51.327 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.697+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 shutdown_connections 2024-02-19T18:49:51.327 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.697+0000 7fd3effed700 1 --2- 172.21.15.77:0/2479641094 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd3e8103d80 0x7fd3e8111820 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:51.327 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.697+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 >> 172.21.15.77:0/2479641094 conn(0x7fd3e80fc580 msgr2=0x7fd3e80fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:51.327 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.697+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 shutdown_connections 2024-02-19T18:49:51.327 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:50.697+0000 7fd3effed700 1 -- 172.21.15.77:0/2479641094 wait complete. 2024-02-19T18:49:51.327 INFO:teuthology.orchestra.run.smithi077.stdout:Restarting the monitor... 2024-02-19T18:49:51.595 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:51 smithi077 systemd[1]: Stopping Ceph mon.smithi077 for 6778a95a-cf57-11ee-95bb-87774f69a715... 2024-02-19T18:49:51.922 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[28873]: 2024-02-19T18:49:51.579+0000 7f525f47d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi077 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T18:49:51.922 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[28873]: 2024-02-19T18:49:51.579+0000 7f525f47d700 -1 mon.smithi077@0(leader) e1 *** Got Signal Terminated *** 2024-02-19T18:49:51.922 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:51 smithi077 podman[29121]: 2024-02-19 18:49:51.693074617 +0000 UTC m=+0.223817687 container died 1cbd3b7e9ebd60a485fedaf6fc74eb4612b4ba2edcad8d19616d17e95239e874 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=quincy-c80e716, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD) 2024-02-19T18:49:52.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 podman[29121]: 2024-02-19 18:49:52.00695656 +0000 UTC m=+0.537699636 container cleanup 1cbd3b7e9ebd60a485fedaf6fc74eb4612b4ba2edcad8d19616d17e95239e874 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=quincy-c80e716, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2) 2024-02-19T18:49:52.230 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 bash[29121]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077 2024-02-19T18:49:52.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 podman[29135]: 2024-02-19 18:49:52.216257429 +0000 UTC m=+0.518058277 container remove 1cbd3b7e9ebd60a485fedaf6fc74eb4612b4ba2edcad8d19616d17e95239e874 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-19T18:49:52.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi077.service: Deactivated successfully. 2024-02-19T18:49:52.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 systemd[1]: Stopped Ceph mon.smithi077 for 6778a95a-cf57-11ee-95bb-87774f69a715. 2024-02-19T18:49:52.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 systemd[1]: Starting Ceph mon.smithi077 for 6778a95a-cf57-11ee-95bb-87774f69a715... 2024-02-19T18:49:52.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 podman[29228]: 2024-02-19 18:49:52.434932206 +0000 UTC m=+0.018184859 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph:quincy 2024-02-19T18:49:52.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 podman[29228]: 2024-02-19 18:49:52.56716954 +0000 UTC m=+0.150422193 container create 7b41d21d0c00d3c4f4079ac54af420c422027c36ed01eb8819ac061178f18628 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 podman[29228]: 2024-02-19 18:49:52.735786376 +0000 UTC m=+0.319039029 container init 7b41d21d0c00d3c4f4079ac54af420c422027c36ed01eb8819ac061178f18628 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, RELEASE=quincy-c80e716, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux ) 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 podman[29228]: 2024-02-19 18:49:52.738623737 +0000 UTC m=+0.321876391 container start 7b41d21d0c00d3c4f4079ac54af420c422027c36ed01eb8819ac061178f18628 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, ceph=True) 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: set uid:gid to 167:167 (ceph:ceph) 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable), process ceph-mon, pid 2 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: pidfile_write: ignore empty --pid-file 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: load: jerasure load: lrc 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: RocksDB version: 6.15.5 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Compile date Feb 19 2024 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: DB SUMMARY 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: DB Session ID: PBFKPK2DOW2C4G23JJPJ 2024-02-19T18:49:52.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: CURRENT file: CURRENT 2024-02-19T18:49:52.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: IDENTITY file: IDENTITY 2024-02-19T18:49:52.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-02-19T18:49:52.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi077/store.db dir, Total Num: 1, files: 000008.sst 2024-02-19T18:49:52.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi077/store.db: 000010.log size: 97180 ; 2024-02-19T18:49:52.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.error_if_exists: 0 2024-02-19T18:49:52.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.create_if_missing: 0 2024-02-19T18:49:52.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.paranoid_checks: 1 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.env: 0x55c5db819a80 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.fs: Posix File System 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.info_log: 0x55c5dc9bfcc0 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_file_opening_threads: 16 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.statistics: (nil) 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.use_fsync: 0 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_log_file_size: 0 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.keep_log_file_num: 1000 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.recycle_log_file_num: 0 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.allow_fallocate: 1 2024-02-19T18:49:52.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.allow_mmap_reads: 0 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.allow_mmap_writes: 0 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.use_direct_reads: 0 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.create_missing_column_families: 0 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.db_log_dir: 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi077/store.db 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.advise_random_on_open: 1 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.db_write_buffer_size: 0 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.write_buffer_manager: 0x55c5dd734b40 2024-02-19T18:49:52.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.rate_limiter: (nil) 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.wal_recovery_mode: 2 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.enable_thread_tracking: 0 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.enable_pipelined_write: 0 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.unordered_write: 0 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-19T18:49:52.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.row_cache: None 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.wal_filter: None 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.allow_ingest_behind: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.preserve_deletes: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.two_write_queues: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.manual_wal_flush: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.atomic_flush: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.log_readahead_size: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.best_efforts_recovery: 0 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-19T18:49:52.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.allow_data_in_errors: 0 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.db_host_id: __hostname__ 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_background_jobs: 2 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_background_compactions: -1 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_subcompactions: 1 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_total_wal_size: 0 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_open_files: -1 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bytes_per_sync: 0 2024-02-19T18:49:52.814 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_readahead_size: 0 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_background_flushes: -1 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Compression algorithms supported: 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: kZSTD supported: 0 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: kXpressCompression supported: 0 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: kLZ4HCCompression supported: 1 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: kZlibCompression supported: 1 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: kSnappyCompression supported: 1 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: kLZ4Compression supported: 1 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: kBZip2Compression supported: 0 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000009 2024-02-19T18:49:52.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.merge_operator: 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_filter: None 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_filter_factory: None 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.sst_partitioner_factory: None 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c5dca14f58) 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: cache_index_and_filter_blocks: 1 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: pin_top_level_index_and_filter: 1 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: index_type: 0 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: data_block_index_type: 0 2024-02-19T18:49:52.816 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: index_shortening: 1 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: hash_index_allow_collision: 1 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: checksum: 1 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: no_block_cache: 0 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_cache: 0x55c5dca47090 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_cache_name: BinnedLRUCache 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_cache_options: 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: capacity : 536870912 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: num_shard_bits : 4 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: strict_capacity_limit : 0 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: high_pri_pool_ratio: 0.000 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_cache_compressed: (nil) 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: persistent_cache: (nil) 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_size: 4096 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_size_deviation: 10 2024-02-19T18:49:52.817 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_restart_interval: 16 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: index_block_restart_interval: 1 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: metadata_block_size: 4096 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: partition_filters: 0 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: use_delta_encoding: 1 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: whole_key_filtering: 1 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: verify_compression: 0 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: read_amp_bytes_per_bit: 0 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: format_version: 4 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: enable_index_compression: 1 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: block_align: 0 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.write_buffer_size: 33554432 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_write_buffer_number: 2 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compression: NoCompression 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bottommost_compression: Disabled 2024-02-19T18:49:52.818 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.prefix_extractor: nullptr 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.num_levels: 7 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compression_opts.level: 32767 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compression_opts.strategy: 0 2024-02-19T18:49:52.819 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compression_opts.enabled: false 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.target_file_size_base: 67108864 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-19T18:49:52.820 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.arena_block_size: 4194304 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.disable_auto_compactions: 0 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-19T18:49:52.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.table_properties_collectors: 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.inplace_update_support: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.bloom_locality: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.max_successive_merges: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.paranoid_file_checks: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.force_consistency_checks: 1 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.report_bg_io_stats: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.ttl: 2592000 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-19T18:49:52.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.enable_blob_files: false 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.min_blob_size: 0 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.blob_file_size: 268435456 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi077/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368592764420, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368592765293, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 91977, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90071, "index_size": 266, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13361, "raw_average_key_size": 49, "raw_value_size": 82771, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 270, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1708368592, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "258dcb22-65a9-4ceb-a329-26837dae1b00", "db_session_id": "PBFKPK2DOW2C4G23JJPJ"}} 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: EVENT_LOG_v1 {"time_micros": 1708368592767927, "job": 1, "event": "recovery_finished"} 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi077/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-19T18:49:52.823 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55c5dc9db180 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: DB pointer 0x55c5dcab0000 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** DB Stats ** 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** Compaction Stats [default] ** 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T18:49:52.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: L0 2/0 91.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Sum 2/0 91.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** Compaction Stats [default] ** 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative compaction: 0.00 GB write, 16.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval compaction: 0.00 GB write, 16.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:49:52.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** Compaction Stats [default] ** 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: L0 2/0 91.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Sum 2/0 91.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** Compaction Stats [default] ** 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.0 0.00 0.00 1 0.001 0 0 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-19T18:49:52.826 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Cumulative compaction: 0.00 GB write, 16.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: starting mon.smithi077 rank 0 at public addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] at bind addrs [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi077 fsid 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077@-1(???) e1 preinit fsid 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077@-1(???).mds e1 new map 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077@-1(???).mds e1 print_map 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: e1 2024-02-19T18:49:52.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: legacy client fscid: -1 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout: No filesystems configured 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-19T18:49:52.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-19T18:49:52.828 INFO:teuthology.orchestra.run.smithi077.stdout:Setting mon public_network to 172.21.0.0/20 2024-02-19T18:49:53.128 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 bash[29228]: 7b41d21d0c00d3c4f4079ac54af420c422027c36ed01eb8819ac061178f18628 2024-02-19T18:49:53.128 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 systemd[1]: Started Ceph mon.smithi077 for 6778a95a-cf57-11ee-95bb-87774f69a715. 2024-02-19T18:49:53.128 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mon.smithi077 is new leader, mons smithi077 in quorum (ranks 0) 2024-02-19T18:49:53.128 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: monmap e1: 1 mons at {smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T18:49:53.128 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: fsmap 2024-02-19T18:49:53.128 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: osdmap e1: 0 total, 0 up, 0 in 2024-02-19T18:49:53.128 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:52 smithi077 ceph-mon[29247]: mgrmap e1: no daemons active 2024-02-19T18:49:53.873 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.209+0000 7f2f55b18700 1 Processor -- start 2024-02-19T18:49:53.873 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.209+0000 7f2f55b18700 1 -- start start 2024-02-19T18:49:53.873 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.209+0000 7f2f55b18700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:53.873 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.209+0000 7f2f55b18700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f501058c0 con 0x7f2f50076c20 2024-02-19T18:49:53.873 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.210+0000 7f2f4f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:53.873 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.210+0000 7f2f4f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58736/0 (socket says 172.21.15.77:58736) 2024-02-19T18:49:53.873 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.210+0000 7f2f4f7fe700 1 -- 172.21.15.77:0/2809286141 learned_addr learned my addr 172.21.15.77:0/2809286141 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:53.873 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.210+0000 7f2f4f7fe700 1 -- 172.21.15.77:0/2809286141 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f50105a00 con 0x7f2f50076c20 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.210+0000 7f2f4f7fe700 1 --2- 172.21.15.77:0/2809286141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50105380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009e50 tx=0x7f2f40009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1775c1c08a2fdcdc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.210+0000 7f2f4effd700 1 -- 172.21.15.77:0/2809286141 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f400092e0 con 0x7f2f50076c20 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.210+0000 7f2f4effd700 1 -- 172.21.15.77:0/2809286141 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2f40012430 con 0x7f2f50076c20 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.210+0000 7f2f4effd700 1 -- 172.21.15.77:0/2809286141 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f400075c0 con 0x7f2f50076c20 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.211+0000 7f2f55b18700 1 -- 172.21.15.77:0/2809286141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 msgr2=0x7f2f50105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.211+0000 7f2f55b18700 1 --2- 172.21.15.77:0/2809286141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50105380 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009e50 tx=0x7f2f40009840 comp rx=0 tx=0).stop 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.211+0000 7f2f55b18700 1 -- 172.21.15.77:0/2809286141 shutdown_connections 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.211+0000 7f2f55b18700 1 --2- 172.21.15.77:0/2809286141 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50105380 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.211+0000 7f2f55b18700 1 -- 172.21.15.77:0/2809286141 >> 172.21.15.77:0/2809286141 conn(0x7f2f500fcdd0 msgr2=0x7f2f500ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.211+0000 7f2f55b18700 1 -- 172.21.15.77:0/2809286141 shutdown_connections 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.211+0000 7f2f55b18700 1 -- 172.21.15.77:0/2809286141 wait complete. 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.211+0000 7f2f55b18700 1 Processor -- start 2024-02-19T18:49:53.874 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f55b18700 1 -- start start 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f55b18700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f55b18700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f50101280 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f4f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f4f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50102e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58738/0 (socket says 172.21.15.77:58738) 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f4f7fe700 1 -- 172.21.15.77:0/3095663349 learned_addr learned my addr 172.21.15.77:0/3095663349 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f4f7fe700 1 -- 172.21.15.77:0/3095663349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f40009ae0 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f4f7fe700 1 --2- 172.21.15.77:0/3095663349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50102e00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2f400127d0 tx=0x7f2f40009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.212+0000 7f2f4cff9700 1 -- 172.21.15.77:0/3095663349 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f400092e0 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.213+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f50101480 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.213+0000 7f2f4cff9700 1 -- 172.21.15.77:0/3095663349 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f2f40007cc0 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.213+0000 7f2f4cff9700 1 -- 172.21.15.77:0/3095663349 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f4001c970 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.213+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f50101970 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.213+0000 7f2f4cff9700 1 -- 172.21.15.77:0/3095663349 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2f40019070 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.213+0000 7f2f4cff9700 1 -- 172.21.15.77:0/3095663349 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f2f40025b40 con 0x7f2f50076c20 2024-02-19T18:49:53.875 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.213+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f5005f5c0 con 0x7f2f50076c20 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.216+0000 7f2f4cff9700 1 -- 172.21.15.77:0/3095663349 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f2f4002a060 con 0x7f2f50076c20 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.281+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f2f5004bed0 con 0x7f2f50076c20 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.283+0000 7f2f4cff9700 1 -- 172.21.15.77:0/3095663349 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f2f40020070 con 0x7f2f50076c20 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.284+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 msgr2=0x7f2f50102e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.284+0000 7f2f55b18700 1 --2- 172.21.15.77:0/3095663349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50102e00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2f400127d0 tx=0x7f2f40009840 comp rx=0 tx=0).stop 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.284+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 shutdown_connections 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.284+0000 7f2f55b18700 1 --2- 172.21.15.77:0/3095663349 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f50076c20 0x7f2f50102e00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.284+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 >> 172.21.15.77:0/3095663349 conn(0x7f2f500fcdd0 msgr2=0x7f2f500fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.284+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 shutdown_connections 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:53.284+0000 7f2f55b18700 1 -- 172.21.15.77:0/3095663349 wait complete. 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:49:53.876 INFO:teuthology.orchestra.run.smithi077.stdout:Creating mgr... 2024-02-19T18:49:53.877 INFO:teuthology.orchestra.run.smithi077.stdout:Verifying port 9283 ... 2024-02-19T18:49:54.037 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mgr.smithi077.yiiqpd 2024-02-19T18:49:54.037 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Failed to reset failed state of unit ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mgr.smithi077.yiiqpd.service: Unit ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mgr.smithi077.yiiqpd.service not loaded. 2024-02-19T18:49:54.182 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-6778a95a-cf57-11ee-95bb-87774f69a715.target.wants/ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mgr.smithi077.yiiqpd.service → /etc/systemd/system/ceph-6778a95a-cf57-11ee-95bb-87774f69a715@.service. 2024-02-19T18:49:54.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:54 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3095663349' entity='client.admin' 2024-02-19T18:49:56.240 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:49:56.240 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-02-19T18:49:56.247 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:49:56.247 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-02-19T18:49:56.247 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-02-19T18:49:56.247 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to enable service . firewalld.service is not available 2024-02-19T18:49:56.255 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:49:56.255 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-02-19T18:49:56.263 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:49:56.263 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-02-19T18:49:56.263 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-02-19T18:49:56.263 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-02-19T18:49:56.263 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr to start... 2024-02-19T18:49:56.263 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr... 2024-02-19T18:49:57.048 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:56 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1201905751' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:49:57.344 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "6778a95a-cf57-11ee-95bb-87774f69a715", 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:49:57.345 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "smithi077" 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:49:57.346 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:49:57.347 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:49:39.682526+0000", 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:49:57.348 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.651+0000 7fc970b25700 1 Processor -- start 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.651+0000 7fc970b25700 1 -- start start 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.652+0000 7fc970b25700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.652+0000 7fc970b25700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c1030c0 con 0x7fc96c102770 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.652+0000 7fc96a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.652+0000 7fc96a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45130/0 (socket says 172.21.15.77:45130) 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.652+0000 7fc96a59c700 1 -- 172.21.15.77:0/836218001 learned_addr learned my addr 172.21.15.77:0/836218001 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.652+0000 7fc96a59c700 1 -- 172.21.15.77:0/836218001 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c103200 con 0x7fc96c102770 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.653+0000 7fc96a59c700 1 --2- 172.21.15.77:0/836218001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c102b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0098e0 tx=0x7fc95c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7be576f43c6c285 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.653+0000 7fc96959a700 1 -- 172.21.15.77:0/836218001 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc95c00b860 con 0x7fc96c102770 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.653+0000 7fc96959a700 1 -- 172.21.15.77:0/836218001 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc95c00be50 con 0x7fc96c102770 2024-02-19T18:49:57.349 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.653+0000 7fc96959a700 1 -- 172.21.15.77:0/836218001 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc95c010380 con 0x7fc96c102770 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.653+0000 7fc970b25700 1 -- 172.21.15.77:0/836218001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 msgr2=0x7fc96c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.653+0000 7fc970b25700 1 --2- 172.21.15.77:0/836218001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c102b80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0098e0 tx=0x7fc95c009250 comp rx=0 tx=0).stop 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 -- 172.21.15.77:0/836218001 shutdown_connections 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 --2- 172.21.15.77:0/836218001 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c102b80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 -- 172.21.15.77:0/836218001 >> 172.21.15.77:0/836218001 conn(0x7fc96c0fded0 msgr2=0x7fc96c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 -- 172.21.15.77:0/836218001 shutdown_connections 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 -- 172.21.15.77:0/836218001 wait complete. 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 Processor -- start 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 -- start start 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.654+0000 7fc970b25700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c111a40 con 0x7fc96c102770 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.655+0000 7fc96a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.655+0000 7fc96a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45146/0 (socket says 172.21.15.77:45146) 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.655+0000 7fc96a59c700 1 -- 172.21.15.77:0/1201905751 learned_addr learned my addr 172.21.15.77:0/1201905751 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:49:57.350 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.655+0000 7fc96a59c700 1 -- 172.21.15.77:0/1201905751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95c0094f0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.656+0000 7fc96a59c700 1 --2- 172.21.15.77:0/1201905751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c111500 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00b440 tx=0x7fc95c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.656+0000 7fc95affd700 1 -- 172.21.15.77:0/1201905751 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc95c0107c0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.656+0000 7fc95affd700 1 -- 172.21.15.77:0/1201905751 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc95c010db0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.656+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96c111c40 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.656+0000 7fc95affd700 1 -- 172.21.15.77:0/1201905751 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc95c01ada0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.656+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96c1128e0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.657+0000 7fc95affd700 1 -- 172.21.15.77:0/1201905751 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc95c024700 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.657+0000 7fc95affd700 1 -- 172.21.15.77:0/1201905751 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc95c02e9f0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.657+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96c05f5c0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.659+0000 7fc95affd700 1 -- 172.21.15.77:0/1201905751 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc95c02ec00 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.731+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc96c1045e0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.731+0000 7fc95affd700 1 -- 172.21.15.77:0/1201905751 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fc95c02eda0 con 0x7fc96c102770 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.732+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 msgr2=0x7fc96c111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.732+0000 7fc970b25700 1 --2- 172.21.15.77:0/1201905751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c111500 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00b440 tx=0x7fc95c009250 comp rx=0 tx=0).stop 2024-02-19T18:49:57.351 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.732+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 shutdown_connections 2024-02-19T18:49:57.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.732+0000 7fc970b25700 1 --2- 172.21.15.77:0/1201905751 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc96c102770 0x7fc96c111500 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:49:57.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.733+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 >> 172.21.15.77:0/1201905751 conn(0x7fc96c0fded0 msgr2=0x7fc96c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:49:57.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.733+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 shutdown_connections 2024-02-19T18:49:57.352 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:56.733+0000 7fc970b25700 1 -- 172.21.15.77:0/1201905751 wait complete. 2024-02-19T18:49:57.352 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (1/15)... 2024-02-19T18:50:00.131 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:49:59 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/829820483' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:50:00.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:50:00.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "6778a95a-cf57-11ee-95bb-87774f69a715", 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "smithi077" 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-02-19T18:50:00.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:50:00.496 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:00.497 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:49:39.682526+0000", 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:00.498 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.752+0000 7fa3af66c700 1 Processor -- start 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.752+0000 7fa3af66c700 1 -- start start 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.752+0000 7fa3af66c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.752+0000 7fa3af66c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a81030c0 con 0x7fa3a8102770 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.752+0000 7fa3ad408700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.752+0000 7fa3ad408700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45148/0 (socket says 172.21.15.77:45148) 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.752+0000 7fa3ad408700 1 -- 172.21.15.77:0/736460513 learned_addr learned my addr 172.21.15.77:0/736460513 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.753+0000 7fa3ad408700 1 -- 172.21.15.77:0/736460513 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a8103200 con 0x7fa3a8102770 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.753+0000 7fa3ad408700 1 --2- 172.21.15.77:0/736460513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8102b80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40098e0 tx=0x7fa3a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428739e36db72f74 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.753+0000 7fa39ffff700 1 -- 172.21.15.77:0/736460513 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a400b860 con 0x7fa3a8102770 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.753+0000 7fa39ffff700 1 -- 172.21.15.77:0/736460513 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa3a400be50 con 0x7fa3a8102770 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.753+0000 7fa39ffff700 1 -- 172.21.15.77:0/736460513 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a4010380 con 0x7fa3a8102770 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.753+0000 7fa3af66c700 1 -- 172.21.15.77:0/736460513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 msgr2=0x7fa3a8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:00.499 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.753+0000 7fa3af66c700 1 --2- 172.21.15.77:0/736460513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8102b80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40098e0 tx=0x7fa3a4009250 comp rx=0 tx=0).stop 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.754+0000 7fa3af66c700 1 -- 172.21.15.77:0/736460513 shutdown_connections 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.754+0000 7fa3af66c700 1 --2- 172.21.15.77:0/736460513 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8102b80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.754+0000 7fa3af66c700 1 -- 172.21.15.77:0/736460513 >> 172.21.15.77:0/736460513 conn(0x7fa3a80fded0 msgr2=0x7fa3a8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.754+0000 7fa3af66c700 1 -- 172.21.15.77:0/736460513 shutdown_connections 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.754+0000 7fa3af66c700 1 -- 172.21.15.77:0/736460513 wait complete. 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.754+0000 7fa3af66c700 1 Processor -- start 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.754+0000 7fa3af66c700 1 -- start start 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa3af66c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa3af66c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3a8111a40 con 0x7fa3a8102770 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa3ad408700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa3ad408700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45150/0 (socket says 172.21.15.77:45150) 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa3ad408700 1 -- 172.21.15.77:0/829820483 learned_addr learned my addr 172.21.15.77:0/829820483 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa3ad408700 1 -- 172.21.15.77:0/829820483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a40094f0 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa3ad408700 1 --2- 172.21.15.77:0/829820483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8111500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400b440 tx=0x7fa3a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa39dffb700 1 -- 172.21.15.77:0/829820483 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a4010760 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa39dffb700 1 -- 172.21.15.77:0/829820483 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa3a4010d50 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa39dffb700 1 -- 172.21.15.77:0/829820483 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3a4024d80 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.755+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3a8111c40 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.756+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3a81128e0 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.756+0000 7fa39dffb700 1 -- 172.21.15.77:0/829820483 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa3a401a6f0 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.756+0000 7fa39dffb700 1 -- 172.21.15.77:0/829820483 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa3a402d9f0 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.756+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3a805f5c0 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.759+0000 7fa39dffb700 1 -- 172.21.15.77:0/829820483 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa3a402dc50 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.830+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa3a81045e0 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.831+0000 7fa39dffb700 1 -- 172.21.15.77:0/829820483 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fa3a402ddf0 con 0x7fa3a8102770 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.832+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 msgr2=0x7fa3a8111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.832+0000 7fa3af66c700 1 --2- 172.21.15.77:0/829820483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8111500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa3a400b440 tx=0x7fa3a4009250 comp rx=0 tx=0).stop 2024-02-19T18:50:00.501 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.832+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 shutdown_connections 2024-02-19T18:50:00.502 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.832+0000 7fa3af66c700 1 --2- 172.21.15.77:0/829820483 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3a8102770 0x7fa3a8111500 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:00.502 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.832+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 >> 172.21.15.77:0/829820483 conn(0x7fa3a80fded0 msgr2=0x7fa3a80fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:00.502 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.832+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 shutdown_connections 2024-02-19T18:50:00.502 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:49:59.832+0000 7fa3af66c700 1 -- 172.21.15.77:0/829820483 wait complete. 2024-02-19T18:50:00.502 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (2/15)... 2024-02-19T18:50:02.883 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: Activating manager daemon smithi077.yiiqpd 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: mgrmap e2: smithi077.yiiqpd(active, starting, since 0.00271842s) 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi077.yiiqpd", "id": "smithi077.yiiqpd"}]: dispatch 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: Manager daemon smithi077.yiiqpd is now available 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/trash_purge_schedule"}]: dispatch 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:02.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:02 smithi077 ceph-mon[29247]: from='mgr.14100 172.21.15.77:0/1037654044' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:03.534 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:50:03.534 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:50:03.534 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "6778a95a-cf57-11ee-95bb-87774f69a715", 2024-02-19T18:50:03.534 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:50:03.534 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:50:03.534 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "smithi077" 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:50:03.535 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:03.536 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:50:03.536 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:50:03.536 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:50:03.543 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:50:03.544 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:49:39.682526+0000", 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.895+0000 7f1d08621700 1 Processor -- start 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.895+0000 7f1d08621700 1 -- start start 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.895+0000 7f1d08621700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d00102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:03.545 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.895+0000 7f1d08621700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d001030c0 con 0x7f1d00102770 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.896+0000 7f1d063bd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d00102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.896+0000 7f1d063bd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d00102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45248/0 (socket says 172.21.15.77:45248) 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.896+0000 7f1d063bd700 1 -- 172.21.15.77:0/4052837948 learned_addr learned my addr 172.21.15.77:0/4052837948 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.896+0000 7f1d063bd700 1 -- 172.21.15.77:0/4052837948 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d00103200 con 0x7f1d00102770 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.896+0000 7f1d063bd700 1 --2- 172.21.15.77:0/4052837948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d00102b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009e50 tx=0x7f1cf0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6cfc33bc935818 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.896+0000 7f1d053bb700 1 -- 172.21.15.77:0/4052837948 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf00092e0 con 0x7f1d00102770 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.896+0000 7f1d053bb700 1 -- 172.21.15.77:0/4052837948 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1cf0012430 con 0x7f1d00102770 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.896+0000 7f1d053bb700 1 -- 172.21.15.77:0/4052837948 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf00075c0 con 0x7f1d00102770 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 -- 172.21.15.77:0/4052837948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 msgr2=0x7f1d00102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 --2- 172.21.15.77:0/4052837948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d00102b80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009e50 tx=0x7f1cf0009840 comp rx=0 tx=0).stop 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 -- 172.21.15.77:0/4052837948 shutdown_connections 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 --2- 172.21.15.77:0/4052837948 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d00102b80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 -- 172.21.15.77:0/4052837948 >> 172.21.15.77:0/4052837948 conn(0x7f1d000fded0 msgr2=0x7f1d00100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:03.546 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 -- 172.21.15.77:0/4052837948 shutdown_connections 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 -- 172.21.15.77:0/4052837948 wait complete. 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 Processor -- start 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 -- start start 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d001115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.897+0000 7f1d08621700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d00111af0 con 0x7f1d00102770 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1d063bd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d001115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1d063bd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d001115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45250/0 (socket says 172.21.15.77:45250) 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1d063bd700 1 -- 172.21.15.77:0/3034635273 learned_addr learned my addr 172.21.15.77:0/3034635273 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1d063bd700 1 -- 172.21.15.77:0/3034635273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf0009ae0 con 0x7f1d00102770 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1d063bd700 1 --2- 172.21.15.77:0/3034635273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d001115b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009ab0 tx=0x7f1cf0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1cfeffd700 1 -- 172.21.15.77:0/3034635273 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf00092e0 con 0x7f1d00102770 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1cfeffd700 1 -- 172.21.15.77:0/3034635273 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1cf0007d10 con 0x7f1d00102770 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d00111cf0 con 0x7f1d00102770 2024-02-19T18:50:03.547 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1cfeffd700 1 -- 172.21.15.77:0/3034635273 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf001c990 con 0x7f1d00102770 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.898+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d00112990 con 0x7f1d00102770 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.899+0000 7f1cfeffd700 1 -- 172.21.15.77:0/3034635273 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f1cf0019070 con 0x7f1d00102770 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.899+0000 7f1cfeffd700 1 -- 172.21.15.77:0/3034635273 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1cf0049a80 con 0x7f1d00102770 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.899+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0005f5c0 con 0x7f1d00102770 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.901+0000 7f1cfeffd700 1 -- 172.21.15.77:0/3034635273 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1cf0017080 con 0x7f1d00102770 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.970+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1d001121e0 con 0x7f1d00102770 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.970+0000 7f1cfeffd700 1 -- 172.21.15.77:0/3034635273 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f1cf0020020 con 0x7f1d00102770 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.971+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 msgr2=0x7f1d001115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.971+0000 7f1d08621700 1 --2- 172.21.15.77:0/3034635273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d001115b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1cf0009ab0 tx=0x7f1cf0009840 comp rx=0 tx=0).stop 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.971+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 shutdown_connections 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.971+0000 7f1d08621700 1 --2- 172.21.15.77:0/3034635273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d00102770 0x7f1d001115b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.971+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 >> 172.21.15.77:0/3034635273 conn(0x7f1d000fded0 msgr2=0x7f1d000ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.971+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 shutdown_connections 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:02.971+0000 7f1d08621700 1 -- 172.21.15.77:0/3034635273 wait complete. 2024-02-19T18:50:03.548 INFO:teuthology.orchestra.run.smithi077.stdout:mgr not available, waiting (3/15)... 2024-02-19T18:50:04.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:03 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3034635273' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:50:04.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:03 smithi077 ceph-mon[29247]: mgrmap e3: smithi077.yiiqpd(active, since 1.00564s) 2024-02-19T18:50:05.939 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:05 smithi077 ceph-mon[29247]: mgrmap e4: smithi077.yiiqpd(active, since 2s) 2024-02-19T18:50:06.758 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:50:06.758 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:50:06.758 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsid": "6778a95a-cf57-11ee-95bb-87774f69a715", 2024-02-19T18:50:06.758 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "health": { 2024-02-19T18:50:06.758 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-19T18:50:06.758 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 0 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "smithi077" 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:06.759 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:06.760 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-19T18:50:06.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-19T18:50:06.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-19T18:50:06.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-19T18:50:06.761 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-19T18:50:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-19T18:50:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-19T18:50:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-19T18:50:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-19T18:50:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-19T18:50:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:06.770 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "restful" 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ], 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:06.771 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "modified": "2024-02-19T18:49:39.682526+0000", 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout }, 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.923+0000 7f4cf72c6700 1 Processor -- start 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.923+0000 7f4cf72c6700 1 -- start start 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.923+0000 7f4cf72c6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0105340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:06.772 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.923+0000 7f4cf72c6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf0105880 con 0x7f4cf00714c0 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.924+0000 7f4cf5062700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0105340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.924+0000 7f4cf5062700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0105340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45256/0 (socket says 172.21.15.77:45256) 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.924+0000 7f4cf5062700 1 -- 172.21.15.77:0/1042126421 learned_addr learned my addr 172.21.15.77:0/1042126421 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.924+0000 7f4cf5062700 1 -- 172.21.15.77:0/1042126421 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf01059c0 con 0x7f4cf00714c0 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.924+0000 7f4cf5062700 1 --2- 172.21.15.77:0/1042126421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0105340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0098e0 tx=0x7f4cec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb214bf2c19c602 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.924+0000 7f4ce7fff700 1 -- 172.21.15.77:0/1042126421 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cec00b860 con 0x7f4cf00714c0 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.924+0000 7f4ce7fff700 1 -- 172.21.15.77:0/1042126421 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4cec00be50 con 0x7f4cf00714c0 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.924+0000 7f4ce7fff700 1 -- 172.21.15.77:0/1042126421 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cec010380 con 0x7f4cf00714c0 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 -- 172.21.15.77:0/1042126421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 msgr2=0x7f4cf0105340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 --2- 172.21.15.77:0/1042126421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0105340 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0098e0 tx=0x7f4cec009250 comp rx=0 tx=0).stop 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 -- 172.21.15.77:0/1042126421 shutdown_connections 2024-02-19T18:50:06.773 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 --2- 172.21.15.77:0/1042126421 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0105340 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 -- 172.21.15.77:0/1042126421 >> 172.21.15.77:0/1042126421 conn(0x7f4cf00fcdd0 msgr2=0x7f4cf00ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 -- 172.21.15.77:0/1042126421 shutdown_connections 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 -- 172.21.15.77:0/1042126421 wait complete. 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 Processor -- start 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.925+0000 7f4cf72c6700 1 -- start start 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf72c6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf72c6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf0111ac0 con 0x7f4cf00714c0 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf5062700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf5062700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45268/0 (socket says 172.21.15.77:45268) 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf5062700 1 -- 172.21.15.77:0/2759071222 learned_addr learned my addr 172.21.15.77:0/2759071222 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf5062700 1 -- 172.21.15.77:0/2759071222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0094f0 con 0x7f4cf00714c0 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf5062700 1 --2- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0111580 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4cec006b00 tx=0x7f4cec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4ce5ffb700 1 -- 172.21.15.77:0/2759071222 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cec0107c0 con 0x7f4cf00714c0 2024-02-19T18:50:06.774 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4ce5ffb700 1 -- 172.21.15.77:0/2759071222 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4cec010db0 con 0x7f4cf00714c0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf0111cc0 con 0x7f4cf00714c0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4ce5ffb700 1 -- 172.21.15.77:0/2759071222 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4cec01acf0 con 0x7f4cf00714c0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.926+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf0112960 con 0x7f4cf00714c0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.927+0000 7f4ce5ffb700 1 -- 172.21.15.77:0/2759071222 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f4cec0193c0 con 0x7f4cf00714c0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.927+0000 7f4ce5ffb700 1 --2- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f4cdc036790 0x7f4cdc038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.927+0000 7f4ce5ffb700 1 -- 172.21.15.77:0/2759071222 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4cec04a7f0 con 0x7f4cf00714c0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.927+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf005f5c0 con 0x7f4cf00714c0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.927+0000 7f4cf4861700 1 --2- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f4cdc036790 0x7f4cdc038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.928+0000 7f4cf4861700 1 --2- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f4cdc036790 0x7f4cdc038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009990 tx=0x7f4ce0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:06.775 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:05.932+0000 7f4ce5ffb700 1 -- 172.21.15.77:0/2759071222 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4cec022930 con 0x7f4cf00714c0 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.190+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4cf0113410 con 0x7f4cf00714c0 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.191+0000 7f4ce5ffb700 1 -- 172.21.15.77:0/2759071222 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f4cec031020 con 0x7f4cf00714c0 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.193+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f4cdc036790 msgr2=0x7f4cdc038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.193+0000 7f4cf72c6700 1 --2- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f4cdc036790 0x7f4cdc038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009990 tx=0x7f4ce0006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.193+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 msgr2=0x7f4cf0111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.193+0000 7f4cf72c6700 1 --2- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0111580 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4cec006b00 tx=0x7f4cec009250 comp rx=0 tx=0).stop 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.193+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 shutdown_connections 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.193+0000 7f4cf72c6700 1 --2- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f4cdc036790 0x7f4cdc038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.193+0000 7f4cf72c6700 1 --2- 172.21.15.77:0/2759071222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4cf00714c0 0x7f4cf0111580 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.193+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 >> 172.21.15.77:0/2759071222 conn(0x7f4cf00fcdd0 msgr2=0x7f4cf00fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:06.776 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.194+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 shutdown_connections 2024-02-19T18:50:06.777 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:06.194+0000 7f4cf72c6700 1 -- 172.21.15.77:0/2759071222 wait complete. 2024-02-19T18:50:06.777 INFO:teuthology.orchestra.run.smithi077.stdout:mgr is available 2024-02-19T18:50:07.054 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:06 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/2759071222' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-19T18:50:07.917 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [global] 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout fsid = 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout [osd] 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-19T18:50:07.918 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.133+0000 7ff04f3ea700 1 Processor -- start 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.134+0000 7ff04f3ea700 1 -- start start 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.134+0000 7ff04f3ea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.134+0000 7ff04f3ea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0481030c0 con 0x7ff048102770 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.134+0000 7ff04d186700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.134+0000 7ff04d186700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50954/0 (socket says 172.21.15.77:50954) 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.134+0000 7ff04d186700 1 -- 172.21.15.77:0/2419190188 learned_addr learned my addr 172.21.15.77:0/2419190188 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.134+0000 7ff04d186700 1 -- 172.21.15.77:0/2419190188 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff048103200 con 0x7ff048102770 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff04d186700 1 --2- 172.21.15.77:0/2419190188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff040009e50 tx=0x7ff040009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e866d7a7ce2ac0e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff03ffff700 1 -- 172.21.15.77:0/2419190188 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0400092e0 con 0x7ff048102770 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff03ffff700 1 -- 172.21.15.77:0/2419190188 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff040012430 con 0x7ff048102770 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff03ffff700 1 -- 172.21.15.77:0/2419190188 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0400075c0 con 0x7ff048102770 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff04f3ea700 1 -- 172.21.15.77:0/2419190188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 msgr2=0x7ff048102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff04f3ea700 1 --2- 172.21.15.77:0/2419190188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff040009e50 tx=0x7ff040009840 comp rx=0 tx=0).stop 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff04f3ea700 1 -- 172.21.15.77:0/2419190188 shutdown_connections 2024-02-19T18:50:07.919 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff04f3ea700 1 --2- 172.21.15.77:0/2419190188 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048102b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff04f3ea700 1 -- 172.21.15.77:0/2419190188 >> 172.21.15.77:0/2419190188 conn(0x7ff0480fded0 msgr2=0x7ff048100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff04f3ea700 1 -- 172.21.15.77:0/2419190188 shutdown_connections 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.135+0000 7ff04f3ea700 1 -- 172.21.15.77:0/2419190188 wait complete. 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.136+0000 7ff04f3ea700 1 Processor -- start 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.136+0000 7ff04f3ea700 1 -- start start 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.136+0000 7ff04f3ea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048111530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.136+0000 7ff04f3ea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff048111a70 con 0x7ff048102770 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.136+0000 7ff04d186700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048111530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.136+0000 7ff04d186700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048111530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50970/0 (socket says 172.21.15.77:50970) 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.136+0000 7ff04d186700 1 -- 172.21.15.77:0/1978199820 learned_addr learned my addr 172.21.15.77:0/1978199820 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.136+0000 7ff04d186700 1 -- 172.21.15.77:0/1978199820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff040009ae0 con 0x7ff048102770 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.137+0000 7ff04d186700 1 --2- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048111530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0400127d0 tx=0x7ff040009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.137+0000 7ff03dffb700 1 -- 172.21.15.77:0/1978199820 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff04000bc30 con 0x7ff048102770 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.137+0000 7ff03dffb700 1 -- 172.21.15.77:0/1978199820 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff04000bd90 con 0x7ff048102770 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.137+0000 7ff03dffb700 1 -- 172.21.15.77:0/1978199820 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff04001c9d0 con 0x7ff048102770 2024-02-19T18:50:07.920 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.137+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff048111c70 con 0x7ff048102770 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.137+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff048112910 con 0x7ff048102770 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.138+0000 7ff03dffb700 1 -- 172.21.15.77:0/1978199820 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7ff040019070 con 0x7ff048102770 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.138+0000 7ff03dffb700 1 --2- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7ff034036740 0x7ff034038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.138+0000 7ff03dffb700 1 -- 172.21.15.77:0/1978199820 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff040049e20 con 0x7ff048102770 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.138+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff04805f5c0 con 0x7ff048102770 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.138+0000 7ff04c985700 1 --2- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7ff034036740 0x7ff034038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.138+0000 7ff04c985700 1 --2- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7ff034036740 0x7ff034038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff038009990 tx=0x7ff038006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.143+0000 7ff03dffb700 1 -- 172.21.15.77:0/1978199820 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff040017080 con 0x7ff048102770 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.328+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7ff04804bed0 con 0x7ff048102770 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.329+0000 7ff03dffb700 1 -- 172.21.15.77:0/1978199820 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7ff040020020 con 0x7ff048102770 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7ff034036740 msgr2=0x7ff034038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 --2- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7ff034036740 0x7ff034038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff038009990 tx=0x7ff038006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 msgr2=0x7ff048111530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 --2- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048111530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff0400127d0 tx=0x7ff040009840 comp rx=0 tx=0).stop 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 shutdown_connections 2024-02-19T18:50:07.921 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 --2- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7ff034036740 0x7ff034038bf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:07.922 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 --2- 172.21.15.77:0/1978199820 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff048102770 0x7ff048111530 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:07.922 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 >> 172.21.15.77:0/1978199820 conn(0x7ff0480fded0 msgr2=0x7ff0480ff580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:07.922 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 shutdown_connections 2024-02-19T18:50:07.922 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:07.331+0000 7ff04f3ea700 1 -- 172.21.15.77:0/1978199820 wait complete. 2024-02-19T18:50:07.922 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling cephadm module... 2024-02-19T18:50:08.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:07 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1978199820' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-19T18:50:09.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:08 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/2500376789' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.279+0000 7f17fa693700 1 Processor -- start 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.279+0000 7f17fa693700 1 -- start start 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.279+0000 7f17fa693700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4103180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.279+0000 7f17fa693700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f41036c0 con 0x7f17f4102d70 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.279+0000 7f17f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4103180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.279+0000 7f17f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4103180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50972/0 (socket says 172.21.15.77:50972) 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.279+0000 7f17f3fff700 1 -- 172.21.15.77:0/2190702872 learned_addr learned my addr 172.21.15.77:0/2190702872 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.279+0000 7f17f3fff700 1 -- 172.21.15.77:0/2190702872 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f4103800 con 0x7f17f4102d70 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17f3fff700 1 --2- 172.21.15.77:0/2190702872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4103180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17e40098e0 tx=0x7f17e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b176ff77312b38b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:09.355 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17f2ffd700 1 -- 172.21.15.77:0/2190702872 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e400b8f0 con 0x7f17f4102d70 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17f2ffd700 1 -- 172.21.15.77:0/2190702872 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f17e4011420 con 0x7f17f4102d70 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17f2ffd700 1 -- 172.21.15.77:0/2190702872 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e4010540 con 0x7f17f4102d70 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17fa693700 1 -- 172.21.15.77:0/2190702872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 msgr2=0x7f17f4103180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17fa693700 1 --2- 172.21.15.77:0/2190702872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4103180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f17e40098e0 tx=0x7f17e4009250 comp rx=0 tx=0).stop 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17fa693700 1 -- 172.21.15.77:0/2190702872 shutdown_connections 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17fa693700 1 --2- 172.21.15.77:0/2190702872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4103180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17fa693700 1 -- 172.21.15.77:0/2190702872 >> 172.21.15.77:0/2190702872 conn(0x7f17f40fc580 msgr2=0x7f17f40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17fa693700 1 -- 172.21.15.77:0/2190702872 shutdown_connections 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.280+0000 7f17fa693700 1 -- 172.21.15.77:0/2190702872 wait complete. 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.281+0000 7f17fa693700 1 Processor -- start 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.281+0000 7f17fa693700 1 -- start start 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.281+0000 7f17fa693700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.281+0000 7f17fa693700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f4113ff0 con 0x7f17f4102d70 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.281+0000 7f17f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:09.356 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.281+0000 7f17f3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4113ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50982/0 (socket says 172.21.15.77:50982) 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.281+0000 7f17f3fff700 1 -- 172.21.15.77:0/2500376789 learned_addr learned my addr 172.21.15.77:0/2500376789 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.282+0000 7f17f3fff700 1 -- 172.21.15.77:0/2500376789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e40094f0 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.282+0000 7f17f3fff700 1 --2- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4113ab0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17e400b440 tx=0x7f17e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.282+0000 7f17f0ff9700 1 -- 172.21.15.77:0/2500376789 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e4010800 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.282+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f41141f0 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.282+0000 7f17f0ff9700 1 -- 172.21.15.77:0/2500376789 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f17e4010df0 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.282+0000 7f17f0ff9700 1 -- 172.21.15.77:0/2500376789 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e401a430 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.282+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f4114e90 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.283+0000 7f17f0ff9700 1 -- 172.21.15.77:0/2500376789 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f17e4010960 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.283+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f405f5c0 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.283+0000 7f17f0ff9700 1 --2- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f17dc036740 0x7f17dc038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.283+0000 7f17f0ff9700 1 -- 172.21.15.77:0/2500376789 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f17e404a760 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.283+0000 7f17f37fe700 1 --2- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f17dc036740 0x7f17dc038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.284+0000 7f17f37fe700 1 --2- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f17dc036740 0x7f17dc038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009990 tx=0x7f17e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.288+0000 7f17f0ff9700 1 -- 172.21.15.77:0/2500376789 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f17e4010c10 con 0x7f17f4102d70 2024-02-19T18:50:09.357 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.523+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f17f404bed0 con 0x7f17f4102d70 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.782+0000 7f17f0ff9700 1 -- 172.21.15.77:0/2500376789 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f17e402dc90 con 0x7f17f4102d70 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.783+0000 7f17f0ff9700 1 -- 172.21.15.77:0/2500376789 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f17e4022b20 con 0x7f17f4102d70 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f17dc036740 msgr2=0x7f17dc038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 --2- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f17dc036740 0x7f17dc038bf0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009990 tx=0x7f17e8006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 msgr2=0x7f17f4113ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 --2- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4113ab0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f17e400b440 tx=0x7f17e4009250 comp rx=0 tx=0).stop 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 shutdown_connections 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 --2- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f17dc036740 0x7f17dc038bf0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 --2- 172.21.15.77:0/2500376789 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17f4102d70 0x7f17f4113ab0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 >> 172.21.15.77:0/2500376789 conn(0x7f17f40fc580 msgr2=0x7f17f40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 shutdown_connections 2024-02-19T18:50:09.358 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:08.786+0000 7f17fa693700 1 -- 172.21.15.77:0/2500376789 wait complete. 2024-02-19T18:50:09.950 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:09 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/2500376789' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-19T18:50:09.950 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:09 smithi077 ceph-mon[29247]: mgrmap e5: smithi077.yiiqpd(active, since 6s) 2024-02-19T18:50:10.597 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:50:10.597 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-19T18:50:10.597 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:50:10.597 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "active_name": "smithi077.yiiqpd", 2024-02-19T18:50:10.597 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.681+0000 7fa10b898700 1 Processor -- start 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa10b898700 1 -- start start 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa10b898700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa10b898700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1041030c0 con 0x7fa104102770 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa109634700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa109634700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51002/0 (socket says 172.21.15.77:51002) 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa109634700 1 -- 172.21.15.77:0/2704407108 learned_addr learned my addr 172.21.15.77:0/2704407108 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa109634700 1 -- 172.21.15.77:0/2704407108 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa104103200 con 0x7fa104102770 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa109634700 1 --2- 172.21.15.77:0/2704407108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa1000098e0 tx=0x7fa100009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85899d250533b037 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.682+0000 7fa0fbfff700 1 -- 172.21.15.77:0/2704407108 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa10000b860 con 0x7fa104102770 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa0fbfff700 1 -- 172.21.15.77:0/2704407108 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa10000be50 con 0x7fa104102770 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa0fbfff700 1 -- 172.21.15.77:0/2704407108 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa100010380 con 0x7fa104102770 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa10b898700 1 -- 172.21.15.77:0/2704407108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 msgr2=0x7fa104102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa10b898700 1 --2- 172.21.15.77:0/2704407108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104102b80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa1000098e0 tx=0x7fa100009250 comp rx=0 tx=0).stop 2024-02-19T18:50:10.598 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa10b898700 1 -- 172.21.15.77:0/2704407108 shutdown_connections 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa10b898700 1 --2- 172.21.15.77:0/2704407108 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104102b80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa10b898700 1 -- 172.21.15.77:0/2704407108 >> 172.21.15.77:0/2704407108 conn(0x7fa1040fded0 msgr2=0x7fa104100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa10b898700 1 -- 172.21.15.77:0/2704407108 shutdown_connections 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.683+0000 7fa10b898700 1 -- 172.21.15.77:0/2704407108 wait complete. 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.684+0000 7fa10b898700 1 Processor -- start 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.684+0000 7fa10b898700 1 -- start start 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.684+0000 7fa10b898700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.684+0000 7fa10b898700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104111a40 con 0x7fa104102770 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.684+0000 7fa109634700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.684+0000 7fa109634700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51010/0 (socket says 172.21.15.77:51010) 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.684+0000 7fa109634700 1 -- 172.21.15.77:0/4075152520 learned_addr learned my addr 172.21.15.77:0/4075152520 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.684+0000 7fa109634700 1 -- 172.21.15.77:0/4075152520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1000094f0 con 0x7fa104102770 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.685+0000 7fa109634700 1 --2- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104111500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa10000b440 tx=0x7fa100009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.685+0000 7fa0f9ffb700 1 -- 172.21.15.77:0/4075152520 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1000107c0 con 0x7fa104102770 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.685+0000 7fa0f9ffb700 1 -- 172.21.15.77:0/4075152520 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa100010db0 con 0x7fa104102770 2024-02-19T18:50:10.599 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.685+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa104111c40 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.685+0000 7fa0f9ffb700 1 -- 172.21.15.77:0/4075152520 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa10001ada0 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.685+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1041128e0 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.686+0000 7fa0f9ffb700 1 -- 172.21.15.77:0/4075152520 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fa100024480 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.686+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10405f5c0 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.686+0000 7fa0f9ffb700 1 --2- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7fa0f0036790 0x7fa0f0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.686+0000 7fa0f9ffb700 1 -- 172.21.15.77:0/4075152520 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa10004bba0 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.686+0000 7fa108e33700 1 -- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7fa0f0036790 msgr2=0x7fa0f0038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2289192383 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.686+0000 7fa108e33700 1 --2- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7fa0f0036790 0x7fa0f0038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.691+0000 7fa0f9ffb700 1 -- 172.21.15.77:0/4075152520 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa10002e440 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.887+0000 7fa108e33700 1 -- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7fa0f0036790 msgr2=0x7fa0f0038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2289192383 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.887+0000 7fa108e33700 1 --2- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7fa0f0036790 0x7fa0f0038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.934+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa104112c20 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.934+0000 7fa0f9ffb700 1 -- 172.21.15.77:0/4075152520 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa10002cb80 con 0x7fa104102770 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7fa0f0036790 msgr2=0x7fa0f0038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:50:10.600 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 --2- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7fa0f0036790 0x7fa0f0038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 msgr2=0x7fa104111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 --2- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104111500 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa10000b440 tx=0x7fa100009250 comp rx=0 tx=0).stop 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 shutdown_connections 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 --2- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7fa0f0036790 0x7fa0f0038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 --2- 172.21.15.77:0/4075152520 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa104102770 0x7fa104111500 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 >> 172.21.15.77:0/4075152520 conn(0x7fa1040fded0 msgr2=0x7fa1040fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 shutdown_connections 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:09.937+0000 7fa10b898700 1 -- 172.21.15.77:0/4075152520 wait complete. 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for the mgr to restart... 2024-02-19T18:50:10.601 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr epoch 5... 2024-02-19T18:50:10.841 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:10 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4075152520' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T18:50:15.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: Active manager daemon smithi077.yiiqpd restarted 2024-02-19T18:50:15.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: Activating manager daemon smithi077.yiiqpd 2024-02-19T18:50:15.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: osdmap e2: 0 total, 0 up, 0 in 2024-02-19T18:50:15.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: mgrmap e6: smithi077.yiiqpd(active, starting, since 0.0032216s) 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi077.yiiqpd", "id": "smithi077.yiiqpd"}]: dispatch 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: Manager daemon smithi077.yiiqpd is now available 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:50:15.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:15 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:50:16.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:16 smithi077 ceph-mon[29247]: Found migration_current of "None". Setting to last migration. 2024-02-19T18:50:16.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:16 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:50:16.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:16 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/trash_purge_schedule"}]: dispatch 2024-02-19T18:50:16.732 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:16 smithi077 ceph-mon[29247]: mgrmap e7: smithi077.yiiqpd(active, since 1.00517s) 2024-02-19T18:50:16.961 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:50:16.961 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-19T18:50:16.961 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-19T18:50:16.961 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.038+0000 7f8226ff8700 1 Processor -- start 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.038+0000 7f8226ff8700 1 -- start start 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.038+0000 7f8226ff8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f82201077e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.038+0000 7f8226ff8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82201031d0 con 0x7f8220076bc0 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.039+0000 7f8224d94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f82201077e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.039+0000 7f8224d94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f82201077e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51026/0 (socket says 172.21.15.77:51026) 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.039+0000 7f8224d94700 1 -- 172.21.15.77:0/236908558 learned_addr learned my addr 172.21.15.77:0/236908558 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.039+0000 7f8224d94700 1 -- 172.21.15.77:0/236908558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8220107d20 con 0x7f8220076bc0 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.039+0000 7f8224d94700 1 --2- 172.21.15.77:0/236908558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f82201077e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82140098e0 tx=0x7f8214009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec20b820ffb2a5e server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.039+0000 7f821f7fe700 1 -- 172.21.15.77:0/236908558 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f821400b860 con 0x7f8220076bc0 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.039+0000 7f821f7fe700 1 -- 172.21.15.77:0/236908558 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f821400be50 con 0x7f8220076bc0 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.039+0000 7f821f7fe700 1 -- 172.21.15.77:0/236908558 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8214010380 con 0x7f8220076bc0 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.040+0000 7f8226ff8700 1 -- 172.21.15.77:0/236908558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 msgr2=0x7f82201077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.040+0000 7f8226ff8700 1 --2- 172.21.15.77:0/236908558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f82201077e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82140098e0 tx=0x7f8214009250 comp rx=0 tx=0).stop 2024-02-19T18:50:16.962 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.040+0000 7f8226ff8700 1 -- 172.21.15.77:0/236908558 shutdown_connections 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.040+0000 7f8226ff8700 1 --2- 172.21.15.77:0/236908558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f82201077e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.040+0000 7f8226ff8700 1 -- 172.21.15.77:0/236908558 >> 172.21.15.77:0/236908558 conn(0x7f82200fcdd0 msgr2=0x7f82200ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.040+0000 7f8226ff8700 1 -- 172.21.15.77:0/236908558 shutdown_connections 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.040+0000 7f8226ff8700 1 -- 172.21.15.77:0/236908558 wait complete. 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.040+0000 7f8226ff8700 1 Processor -- start 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.041+0000 7f8226ff8700 1 -- start start 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.041+0000 7f8226ff8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f8220113720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.041+0000 7f8226ff8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8220113c60 con 0x7f8220076bc0 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.041+0000 7f8224d94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f8220113720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.041+0000 7f8224d94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f8220113720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51034/0 (socket says 172.21.15.77:51034) 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.041+0000 7f8224d94700 1 -- 172.21.15.77:0/1117387514 learned_addr learned my addr 172.21.15.77:0/1117387514 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.041+0000 7f8224d94700 1 -- 172.21.15.77:0/1117387514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82140094f0 con 0x7f8220076bc0 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.041+0000 7f8224d94700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f8220113720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f821400b440 tx=0x7f8214009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.042+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f82140107c0 con 0x7f8220076bc0 2024-02-19T18:50:16.963 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.042+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8214010db0 con 0x7f8220076bc0 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.042+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8220113e60 con 0x7f8220076bc0 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.042+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8220114b00 con 0x7f8220076bc0 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.042+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f821401ad30 con 0x7f8220076bc0 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.042+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f8214024490 con 0x7f8220076bc0 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.043+0000 7f821d7fa700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 0x7f8210038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.043+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 --> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8210039350 con 0x7f8210036790 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.043+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f821404ab90 con 0x7f8220076bc0 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.043+0000 7f821ffff700 1 -- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 msgr2=0x7f8210038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2289192383 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.043+0000 7f821ffff700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 0x7f8210038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.244+0000 7f821ffff700 1 -- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 msgr2=0x7f8210038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2289192383 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.244+0000 7f821ffff700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 0x7f8210038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.645+0000 7f821ffff700 1 -- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 msgr2=0x7f8210038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2289192383 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:11.645+0000 7f821ffff700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 0x7f8210038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T18:50:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:12.446+0000 7f821ffff700 1 -- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 msgr2=0x7f8210038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2289192383 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:12.446+0000 7f821ffff700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 0x7f8210038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:14.047+0000 7f821ffff700 1 -- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 msgr2=0x7f8210038c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/2289192383 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:14.047+0000 7f821ffff700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 0x7f8210038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:15.315+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7f8214024a10 con 0x7f8220076bc0 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:15.315+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 msgr2=0x7f8210038c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:15.315+0000 7f821d7fa700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 0x7f8210038c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.316+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f821402c390 con 0x7f8220076bc0 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.316+0000 7f821d7fa700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f8210039fc0 0x7f821003c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.316+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8210039350 con 0x7f8210039fc0 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.316+0000 7f821ffff700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f8210039fc0 0x7f821003c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.317+0000 7f821ffff700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f8210039fc0 0x7f821003c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f820c0039d0 tx=0x7f820c009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.318+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f8210039350 con 0x7f8210039fc0 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.323+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f82040025f0 con 0x7f8210039fc0 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f821d7fa700 1 -- 172.21.15.77:0/1117387514 <== mgr.14120 v2:172.21.15.77:6800/1028404574 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f82040025f0 con 0x7f8210039fc0 2024-02-19T18:50:16.965 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f8210039fc0 msgr2=0x7f821003c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f8210039fc0 0x7f821003c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f820c0039d0 tx=0x7f820c009350 comp rx=0 tx=0).stop 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 msgr2=0x7f8220113720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f8220113720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f821400b440 tx=0x7f8214009250 comp rx=0 tx=0).stop 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 shutdown_connections 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f8210039fc0 0x7f821003c3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8220076bc0 0x7f8220113720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 --2- 172.21.15.77:0/1117387514 >> [v2:172.21.15.77:6800/2289192383,v1:172.21.15.77:6801/2289192383] conn(0x7f8210036790 0x7f8210038c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 >> 172.21.15.77:0/1117387514 conn(0x7f82200fcdd0 msgr2=0x7f82200fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 shutdown_connections 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:16.324+0000 7f8226ff8700 1 -- 172.21.15.77:0/1117387514 wait complete. 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:mgr epoch 5 is available 2024-02-19T18:50:16.966 INFO:teuthology.orchestra.run.smithi077.stdout:Setting orchestrator backend to cephadm... 2024-02-19T18:50:17.975 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T18:50:17.975 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T18:50:17.975 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: [19/Feb/2024:18:50:16] ENGINE Bus STARTING 2024-02-19T18:50:17.976 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: [19/Feb/2024:18:50:16] ENGINE Serving on https://172.21.15.77:7150 2024-02-19T18:50:17.976 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: [19/Feb/2024:18:50:16] ENGINE Bus STARTED 2024-02-19T18:50:17.976 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:17.976 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:50:17.976 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:17.976 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:17 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f5855306700 1 Processor -- start 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f5855306700 1 -- start start 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f5855306700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850071480 0x7f5850105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f5855306700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58501058c0 con 0x7f5850071480 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f584effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850071480 0x7f5850105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f584effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850071480 0x7f5850105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56412/0 (socket says 172.21.15.77:56412) 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f584effd700 1 -- 172.21.15.77:0/2560620661 learned_addr learned my addr 172.21.15.77:0/2560620661 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f584effd700 1 -- 172.21.15.77:0/2560620661 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5850105a00 con 0x7f5850071480 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f584effd700 1 --2- 172.21.15.77:0/2560620661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850071480 0x7f5850105380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f58400098e0 tx=0x7f5840009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a52e514076901 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.395+0000 7f584dffb700 1 -- 172.21.15.77:0/2560620661 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f584000b860 con 0x7f5850071480 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f584dffb700 1 -- 172.21.15.77:0/2560620661 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f584000be50 con 0x7f5850071480 2024-02-19T18:50:18.242 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f584dffb700 1 -- 172.21.15.77:0/2560620661 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5840010380 con 0x7f5850071480 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f5855306700 1 -- 172.21.15.77:0/2560620661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850071480 msgr2=0x7f5850105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f5855306700 1 --2- 172.21.15.77:0/2560620661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850071480 0x7f5850105380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f58400098e0 tx=0x7f5840009250 comp rx=0 tx=0).stop 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f5855306700 1 -- 172.21.15.77:0/2560620661 shutdown_connections 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f5855306700 1 --2- 172.21.15.77:0/2560620661 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850071480 0x7f5850105380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f5855306700 1 -- 172.21.15.77:0/2560620661 >> 172.21.15.77:0/2560620661 conn(0x7f58500fcdd0 msgr2=0x7f58500ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f5855306700 1 -- 172.21.15.77:0/2560620661 shutdown_connections 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.396+0000 7f5855306700 1 -- 172.21.15.77:0/2560620661 wait complete. 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.397+0000 7f5855306700 1 Processor -- start 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.397+0000 7f5855306700 1 -- start start 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.397+0000 7f5855306700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850111700 0x7f5850111b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.397+0000 7f5855306700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5850112050 con 0x7f5850111700 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.397+0000 7f584effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850111700 0x7f5850111b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.397+0000 7f584effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850111700 0x7f5850111b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56414/0 (socket says 172.21.15.77:56414) 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.397+0000 7f584effd700 1 -- 172.21.15.77:0/2234525499 learned_addr learned my addr 172.21.15.77:0/2234525499 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:18.243 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.397+0000 7f584effd700 1 -- 172.21.15.77:0/2234525499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58400094f0 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.398+0000 7f584effd700 1 --2- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850111700 0x7f5850111b10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f584000b440 tx=0x7f5840009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.398+0000 7f58377fe700 1 -- 172.21.15.77:0/2234525499 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5840010600 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.398+0000 7f58377fe700 1 -- 172.21.15.77:0/2234525499 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f584001f070 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.398+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5850112250 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.398+0000 7f58377fe700 1 -- 172.21.15.77:0/2234525499 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5840026b90 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.398+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5850112d60 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.398+0000 7f58377fe700 1 -- 172.21.15.77:0/2234525499 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7f5840026cf0 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.399+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585004bed0 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.399+0000 7f58377fe700 1 --2- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f58380366b0 0x7f5838038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.399+0000 7f58377fe700 1 -- 172.21.15.77:0/2234525499 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5840021070 con 0x7f5850111700 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.399+0000 7f584e7fc700 1 --2- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f58380366b0 0x7f5838038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:18.244 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.399+0000 7f584e7fc700 1 --2- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f58380366b0 0x7f5838038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5844009990 tx=0x7f5844006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.403+0000 7f58377fe700 1 -- 172.21.15.77:0/2234525499 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5840024970 con 0x7f5850111700 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.596+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f5850112fa0 con 0x7f58380366b0 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.603+0000 7f58377fe700 1 -- 172.21.15.77:0/2234525499 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5850112fa0 con 0x7f58380366b0 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f58380366b0 msgr2=0x7f5838038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 --2- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f58380366b0 0x7f5838038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5844009990 tx=0x7f5844006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850111700 msgr2=0x7f5850111b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 --2- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850111700 0x7f5850111b10 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f584000b440 tx=0x7f5840009250 comp rx=0 tx=0).stop 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 shutdown_connections 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 --2- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f58380366b0 0x7f5838038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 --2- 172.21.15.77:0/2234525499 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5850111700 0x7f5850111b10 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 >> 172.21.15.77:0/2234525499 conn(0x7f58500fcdd0 msgr2=0x7f58500fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 shutdown_connections 2024-02-19T18:50:18.245 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:17.605+0000 7f5855306700 1 -- 172.21.15.77:0/2234525499 wait complete. 2024-02-19T18:50:18.799 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:18 smithi077 ceph-mon[29247]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:18.799 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:18 smithi077 ceph-mon[29247]: mgrmap e8: smithi077.yiiqpd(active, since 2s) 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.593+0000 7f559f3ac700 1 Processor -- start 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.593+0000 7f559f3ac700 1 -- start start 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.593+0000 7f559f3ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598102b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.593+0000 7f559f3ac700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5598103050 con 0x7f5598102700 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.593+0000 7f559d148700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.593+0000 7f559d148700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598102b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56428/0 (socket says 172.21.15.77:56428) 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.593+0000 7f559d148700 1 -- 172.21.15.77:0/2103422380 learned_addr learned my addr 172.21.15.77:0/2103422380 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.593+0000 7f559d148700 1 -- 172.21.15.77:0/2103422380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5598103190 con 0x7f5598102700 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f559d148700 1 --2- 172.21.15.77:0/2103422380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598102b10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f55940098e0 tx=0x7f5594009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67cd571e14f13b64 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f558ffff700 1 -- 172.21.15.77:0/2103422380 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f559400b860 con 0x7f5598102700 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f558ffff700 1 -- 172.21.15.77:0/2103422380 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f559400be50 con 0x7f5598102700 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f558ffff700 1 -- 172.21.15.77:0/2103422380 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5594010380 con 0x7f5598102700 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f559f3ac700 1 -- 172.21.15.77:0/2103422380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 msgr2=0x7f5598102b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f559f3ac700 1 --2- 172.21.15.77:0/2103422380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598102b10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f55940098e0 tx=0x7f5594009250 comp rx=0 tx=0).stop 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f559f3ac700 1 -- 172.21.15.77:0/2103422380 shutdown_connections 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f559f3ac700 1 --2- 172.21.15.77:0/2103422380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598102b10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f559f3ac700 1 -- 172.21.15.77:0/2103422380 >> 172.21.15.77:0/2103422380 conn(0x7f55980fde60 msgr2=0x7f5598100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f559f3ac700 1 -- 172.21.15.77:0/2103422380 shutdown_connections 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.594+0000 7f559f3ac700 1 -- 172.21.15.77:0/2103422380 wait complete. 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559f3ac700 1 Processor -- start 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559f3ac700 1 -- start start 2024-02-19T18:50:19.374 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559f3ac700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598111480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559f3ac700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55981119c0 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559d148700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598111480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559d148700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598111480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56442/0 (socket says 172.21.15.77:56442) 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559d148700 1 -- 172.21.15.77:0/3523876730 learned_addr learned my addr 172.21.15.77:0/3523876730 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559d148700 1 -- 172.21.15.77:0/3523876730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55940094f0 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.595+0000 7f559d148700 1 --2- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598111480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f559400b440 tx=0x7f5594009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.596+0000 7f558dffb700 1 -- 172.21.15.77:0/3523876730 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55940107c0 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.596+0000 7f558dffb700 1 -- 172.21.15.77:0/3523876730 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f5594010db0 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.596+0000 7f558dffb700 1 -- 172.21.15.77:0/3523876730 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f559401ada0 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.596+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5598111bc0 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.596+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5598112860 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.597+0000 7f558dffb700 1 -- 172.21.15.77:0/3523876730 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5594010920 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.597+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f559805f5c0 con 0x7f5598102700 2024-02-19T18:50:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.597+0000 7f558dffb700 1 --2- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f55840367e0 0x7f5584038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.597+0000 7f558dffb700 1 -- 172.21.15.77:0/3523876730 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f559404acf0 con 0x7f5598102700 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.597+0000 7f559c947700 1 --2- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f55840367e0 0x7f5584038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.597+0000 7f559c947700 1 --2- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f55840367e0 0x7f5584038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5588009990 tx=0x7f5588006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.602+0000 7f558dffb700 1 -- 172.21.15.77:0/3523876730 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5594018940 con 0x7f5598102700 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.784+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f55980ff350 con 0x7f55840367e0 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.784+0000 7f558dffb700 1 -- 172.21.15.77:0/3523876730 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f55980ff350 con 0x7f55840367e0 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f55840367e0 msgr2=0x7f5584038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 --2- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f55840367e0 0x7f5584038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5588009990 tx=0x7f5588006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 msgr2=0x7f5598111480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 --2- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598111480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f559400b440 tx=0x7f5594009250 comp rx=0 tx=0).stop 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 shutdown_connections 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 --2- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f55840367e0 0x7f5584038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 --2- 172.21.15.77:0/3523876730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5598102700 0x7f5598111480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:19.376 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 >> 172.21.15.77:0/3523876730 conn(0x7f55980fde60 msgr2=0x7f55980fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:19.377 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 shutdown_connections 2024-02-19T18:50:19.377 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:18.786+0000 7f559f3ac700 1 -- 172.21.15.77:0/3523876730 wait complete. 2024-02-19T18:50:19.377 INFO:teuthology.orchestra.run.smithi077.stdout:Generating ssh key... 2024-02-19T18:50:19.980 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:19 smithi077 ceph-mon[29247]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:20.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.773+0000 7f4a10c3e700 1 Processor -- start 2024-02-19T18:50:20.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.773+0000 7f4a10c3e700 1 -- start start 2024-02-19T18:50:20.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.773+0000 7f4a10c3e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:20.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.773+0000 7f4a10c3e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c1030c0 con 0x7f4a0c102770 2024-02-19T18:50:20.662 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.773+0000 7f4a0a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.773+0000 7f4a0a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56452/0 (socket says 172.21.15.77:56452) 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.773+0000 7f4a0a59c700 1 -- 172.21.15.77:0/3521207313 learned_addr learned my addr 172.21.15.77:0/3521207313 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.774+0000 7f4a0a59c700 1 -- 172.21.15.77:0/3521207313 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c103200 con 0x7f4a0c102770 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.774+0000 7f4a0a59c700 1 --2- 172.21.15.77:0/3521207313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c102b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0098e0 tx=0x7f49fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a5dfb4a36e6cb8 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.774+0000 7f4a0959a700 1 -- 172.21.15.77:0/3521207313 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49fc00b860 con 0x7f4a0c102770 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.774+0000 7f4a0959a700 1 -- 172.21.15.77:0/3521207313 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f49fc00be50 con 0x7f4a0c102770 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.774+0000 7f4a0959a700 1 -- 172.21.15.77:0/3521207313 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49fc010380 con 0x7f4a0c102770 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 -- 172.21.15.77:0/3521207313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 msgr2=0x7f4a0c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 --2- 172.21.15.77:0/3521207313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c102b80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f49fc0098e0 tx=0x7f49fc009250 comp rx=0 tx=0).stop 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 -- 172.21.15.77:0/3521207313 shutdown_connections 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 --2- 172.21.15.77:0/3521207313 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c102b80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 -- 172.21.15.77:0/3521207313 >> 172.21.15.77:0/3521207313 conn(0x7f4a0c0fded0 msgr2=0x7f4a0c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 -- 172.21.15.77:0/3521207313 shutdown_connections 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 -- 172.21.15.77:0/3521207313 wait complete. 2024-02-19T18:50:20.663 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 Processor -- start 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 -- start start 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c078d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.775+0000 7f4a10c3e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a0c077190 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f4a0a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c078d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f4a0a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c078d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56462/0 (socket says 172.21.15.77:56462) 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f4a0a59c700 1 -- 172.21.15.77:0/499184562 learned_addr learned my addr 172.21.15.77:0/499184562 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f4a0a59c700 1 -- 172.21.15.77:0/499184562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc0094f0 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f4a0a59c700 1 --2- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c078d10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00b440 tx=0x7f49fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f49faffd700 1 -- 172.21.15.77:0/499184562 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49fc010760 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f49faffd700 1 -- 172.21.15.77:0/499184562 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f49fc010d50 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f49faffd700 1 -- 172.21.15.77:0/499184562 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49fc01ae20 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a0c077390 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.776+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a0c077820 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.777+0000 7f49faffd700 1 -- 172.21.15.77:0/499184562 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f49fc02e440 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.777+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a0c04bed0 con 0x7f4a0c102770 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.777+0000 7f49faffd700 1 --2- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f49f4036770 0x7f49f4038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:20.664 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.777+0000 7f49faffd700 1 -- 172.21.15.77:0/499184562 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f49fc04a860 con 0x7f4a0c102770 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.777+0000 7f4a09d9b700 1 --2- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f49f4036770 0x7f49f4038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.778+0000 7f4a09d9b700 1 --2- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f49f4036770 0x7f49f4038c20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009990 tx=0x7f4a00006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.783+0000 7f49faffd700 1 -- 172.21.15.77:0/499184562 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f49fc018710 con 0x7f4a0c102770 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:19.964+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a0c077fa0 con 0x7f49f4036770 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.074+0000 7f49faffd700 1 -- 172.21.15.77:0/499184562 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4a0c077fa0 con 0x7f49f4036770 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f49f4036770 msgr2=0x7f49f4038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 --2- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f49f4036770 0x7f49f4038c20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009990 tx=0x7f4a00006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 msgr2=0x7f4a0c078d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 --2- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c078d10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f49fc00b440 tx=0x7f49fc009250 comp rx=0 tx=0).stop 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 shutdown_connections 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 --2- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f49f4036770 0x7f49f4038c20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 --2- 172.21.15.77:0/499184562 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4a0c102770 0x7f4a0c078d10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 >> 172.21.15.77:0/499184562 conn(0x7f4a0c0fded0 msgr2=0x7f4a0c0ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:20.665 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 shutdown_connections 2024-02-19T18:50:20.666 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:20.076+0000 7f4a10c3e700 1 -- 172.21.15.77:0/499184562 wait complete. 2024-02-19T18:50:21.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:21 smithi077 ceph-mon[29247]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:21.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:21 smithi077 ceph-mon[29247]: Generating ssh key... 2024-02-19T18:50:21.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:21 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:21.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:21 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:21.810 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIFJBfDNK+jtfEcX4T56qJQTp5o+vjm8mmcglfds0U13RN3MyQmfQeym98bfMfZz9wav42zhWN0mHDojHvJBXYScB9RE0brQ8qFKypWQrGonKXYs2+TqGGJQw2cE7E7z5E25EnkiIBPbPlNwL6c0FBzOzA4CoRuc5mgpc3VlZANyq5QHWIry6magvLY5Cwq/ZfK39Ebd+Mcz73dlgkff4tx6WYBmFRbzNL/VJiruSdgufsuoxQPD/y64Lj/SYABR30mBUES678SA+2uDSx7sZAtKgE30q9Zzos3AXZDE2HKaNtY17FjMk6dBAljjBJD4oDLSjEss7wNXP7bLHbofN5qV9IWbWKgSObDZsgom7vv+F509/xhYgVQQhAEOdrzjLg1MpiHhCIqYUo/aSjNHvskqZY26b7Lt0eNb/+VTSayoVOdBcUKRUppBSRNBqyAdV6J1y6NckbIBddhKStPmcYlI2Ic9f2FhcKnzIQSC2g9tNHlkM6jbkc9dvr1UWzPLs= ceph-6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:50:21.810 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.014+0000 7fbbccc97700 1 Processor -- start 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.014+0000 7fbbccc97700 1 -- start start 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.014+0000 7fbbccc97700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.014+0000 7fbbccc97700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81030c0 con 0x7fbbc8102770 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.015+0000 7fbbc659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.015+0000 7fbbc659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56466/0 (socket says 172.21.15.77:56466) 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.015+0000 7fbbc659c700 1 -- 172.21.15.77:0/2254455834 learned_addr learned my addr 172.21.15.77:0/2254455834 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.015+0000 7fbbc659c700 1 -- 172.21.15.77:0/2254455834 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc8103200 con 0x7fbbc8102770 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.015+0000 7fbbc659c700 1 --2- 172.21.15.77:0/2254455834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8102b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80098e0 tx=0x7fbbb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd2261e6c40d5790 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.015+0000 7fbbc559a700 1 -- 172.21.15.77:0/2254455834 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb800b860 con 0x7fbbc8102770 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.015+0000 7fbbc559a700 1 -- 172.21.15.77:0/2254455834 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbbb800be50 con 0x7fbbc8102770 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.015+0000 7fbbc559a700 1 -- 172.21.15.77:0/2254455834 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb8010380 con 0x7fbbc8102770 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 -- 172.21.15.77:0/2254455834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 msgr2=0x7fbbc8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 --2- 172.21.15.77:0/2254455834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8102b80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbbb80098e0 tx=0x7fbbb8009250 comp rx=0 tx=0).stop 2024-02-19T18:50:21.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 -- 172.21.15.77:0/2254455834 shutdown_connections 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 --2- 172.21.15.77:0/2254455834 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8102b80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 -- 172.21.15.77:0/2254455834 >> 172.21.15.77:0/2254455834 conn(0x7fbbc80fded0 msgr2=0x7fbbc8100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 -- 172.21.15.77:0/2254455834 shutdown_connections 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 -- 172.21.15.77:0/2254455834 wait complete. 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 Processor -- start 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.016+0000 7fbbccc97700 1 -- start start 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbccc97700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbccc97700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8111a40 con 0x7fbbc8102770 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbc659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbc659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8111500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56468/0 (socket says 172.21.15.77:56468) 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbc659c700 1 -- 172.21.15.77:0/3247819496 learned_addr learned my addr 172.21.15.77:0/3247819496 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbc659c700 1 -- 172.21.15.77:0/3247819496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb80094f0 con 0x7fbbc8102770 2024-02-19T18:50:21.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbc659c700 1 --2- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8111500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800b440 tx=0x7fbbb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbb6ffd700 1 -- 172.21.15.77:0/3247819496 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb80107c0 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbb6ffd700 1 -- 172.21.15.77:0/3247819496 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbbb8010db0 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbb6ffd700 1 -- 172.21.15.77:0/3247819496 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbb801ada0 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc8111c40 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.017+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc81128e0 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.018+0000 7fbbb6ffd700 1 -- 172.21.15.77:0/3247819496 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fbbb8024480 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.018+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc805f5c0 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.018+0000 7fbbb6ffd700 1 --2- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fbbb0036790 0x7fbbb0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.019+0000 7fbbb6ffd700 1 -- 172.21.15.77:0/3247819496 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbbb804ae40 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.019+0000 7fbbc5d9b700 1 --2- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fbbb0036790 0x7fbbb0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.019+0000 7fbbc5d9b700 1 --2- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fbbb0036790 0x7fbbb0038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009990 tx=0x7fbbbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.023+0000 7fbbb6ffd700 1 -- 172.21.15.77:0/3247819496 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbbb8018350 con 0x7fbbc8102770 2024-02-19T18:50:21.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.208+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbc80ff380 con 0x7fbbb0036790 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.209+0000 7fbbb6ffd700 1 -- 172.21.15.77:0/3247819496 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fbbc80ff380 con 0x7fbbb0036790 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.210+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fbbb0036790 msgr2=0x7fbbb0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.210+0000 7fbbccc97700 1 --2- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fbbb0036790 0x7fbbb0038c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009990 tx=0x7fbbbc006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.210+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 msgr2=0x7fbbc8111500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.210+0000 7fbbccc97700 1 --2- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8111500 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800b440 tx=0x7fbbb8009250 comp rx=0 tx=0).stop 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.210+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 shutdown_connections 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.210+0000 7fbbccc97700 1 --2- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fbbb0036790 0x7fbbb0038c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.210+0000 7fbbccc97700 1 --2- 172.21.15.77:0/3247819496 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbbc8102770 0x7fbbc8111500 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.210+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 >> 172.21.15.77:0/3247819496 conn(0x7fbbc80fded0 msgr2=0x7fbbc80fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.211+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 shutdown_connections 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:21.211+0000 7fbbccc97700 1 -- 172.21.15.77:0/3247819496 wait complete. 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:Adding key to root@localhost authorized_keys... 2024-02-19T18:50:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:Adding host smithi077... 2024-02-19T18:50:22.247 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:22 smithi077 ceph-mon[29247]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:24.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:23 smithi077 ceph-mon[29247]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:25.028 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:24 smithi077 ceph-mon[29247]: Deploying cephadm binary to smithi077 2024-02-19T18:50:25.028 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:24 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:25.028 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:24 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:50:25.035 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Added host 'smithi077' with addr '172.21.15.77' 2024-02-19T18:50:25.035 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.249+0000 7f6da3bd9700 1 Processor -- start 2024-02-19T18:50:25.035 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.249+0000 7f6da3bd9700 1 -- start start 2024-02-19T18:50:25.035 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.249+0000 7f6da3bd9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:25.035 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.249+0000 7f6da3bd9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c1014f0 con 0x7f6d9c100ba0 2024-02-19T18:50:25.035 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.249+0000 7f6da1975700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:25.035 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.249+0000 7f6da1975700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56476/0 (socket says 172.21.15.77:56476) 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.249+0000 7f6da1975700 1 -- 172.21.15.77:0/4229473799 learned_addr learned my addr 172.21.15.77:0/4229473799 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.250+0000 7f6da1975700 1 -- 172.21.15.77:0/4229473799 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c101630 con 0x7f6d9c100ba0 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.250+0000 7f6da1975700 1 --2- 172.21.15.77:0/4229473799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c100fb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009e50 tx=0x7f6d94009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8e515e38c6f7f4a server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.250+0000 7f6da0973700 1 -- 172.21.15.77:0/4229473799 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d940092e0 con 0x7f6d9c100ba0 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.250+0000 7f6da0973700 1 -- 172.21.15.77:0/4229473799 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6d94012430 con 0x7f6d9c100ba0 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.250+0000 7f6da0973700 1 -- 172.21.15.77:0/4229473799 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d940075c0 con 0x7f6d9c100ba0 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 -- 172.21.15.77:0/4229473799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 msgr2=0x7f6d9c100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 --2- 172.21.15.77:0/4229473799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c100fb0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6d94009e50 tx=0x7f6d94009840 comp rx=0 tx=0).stop 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 -- 172.21.15.77:0/4229473799 shutdown_connections 2024-02-19T18:50:25.036 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 --2- 172.21.15.77:0/4229473799 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c100fb0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 -- 172.21.15.77:0/4229473799 >> 172.21.15.77:0/4229473799 conn(0x7f6d9c0fc580 msgr2=0x7f6d9c0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 -- 172.21.15.77:0/4229473799 shutdown_connections 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 -- 172.21.15.77:0/4229473799 wait complete. 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 Processor -- start 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 -- start start 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c111870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da3bd9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c111db0 con 0x7f6d9c100ba0 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.251+0000 7f6da1975700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6da1975700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c111870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56480/0 (socket says 172.21.15.77:56480) 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6da1975700 1 -- 172.21.15.77:0/734504344 learned_addr learned my addr 172.21.15.77:0/734504344 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6da1975700 1 -- 172.21.15.77:0/734504344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d94009ae0 con 0x7f6d9c100ba0 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6da1975700 1 --2- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c111870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d940127d0 tx=0x7f6d94009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6d927fc700 1 -- 172.21.15.77:0/734504344 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d9400bc30 con 0x7f6d9c100ba0 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c111fb0 con 0x7f6d9c100ba0 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6d927fc700 1 -- 172.21.15.77:0/734504344 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6d9400bd90 con 0x7f6d9c100ba0 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6d927fc700 1 -- 172.21.15.77:0/734504344 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d9401c9d0 con 0x7f6d9c100ba0 2024-02-19T18:50:25.037 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.252+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c112c50 con 0x7f6d9c100ba0 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.253+0000 7f6d927fc700 1 -- 172.21.15.77:0/734504344 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6d94019070 con 0x7f6d9c100ba0 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.253+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c05f5c0 con 0x7f6d9c100ba0 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.253+0000 7f6d927fc700 1 --2- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f6d88036790 0x7f6d88038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.253+0000 7f6d927fc700 1 -- 172.21.15.77:0/734504344 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6d94049e70 con 0x7f6d9c100ba0 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.253+0000 7f6da1174700 1 --2- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f6d88036790 0x7f6d88038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.254+0000 7f6da1174700 1 --2- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f6d88036790 0x7f6d88038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009990 tx=0x7f6d8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.258+0000 7f6d927fc700 1 -- 172.21.15.77:0/734504344 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6d94007880 con 0x7f6d9c100ba0 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:22.443+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi077", "addr": "172.21.15.77", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d9c1069c0 con 0x7f6d88036790 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.427+0000 7f6d927fc700 1 -- 172.21.15.77:0/734504344 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f6d8c009e60 con 0x7f6d88036790 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f6d88036790 msgr2=0x7f6d88038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 --2- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f6d88036790 0x7f6d88038c40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c009990 tx=0x7f6d8c006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 msgr2=0x7f6d9c111870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 --2- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c111870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6d940127d0 tx=0x7f6d94009840 comp rx=0 tx=0).stop 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 shutdown_connections 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 --2- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f6d88036790 0x7f6d88038c40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:25.038 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 --2- 172.21.15.77:0/734504344 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6d9c100ba0 0x7f6d9c111870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 >> 172.21.15.77:0/734504344 conn(0x7f6d9c0fc580 msgr2=0x7f6d9c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 shutdown_connections 2024-02-19T18:50:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:24.429+0000 7f6da3bd9700 1 -- 172.21.15.77:0/734504344 wait complete. 2024-02-19T18:50:25.039 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying mon service with default placement... 2024-02-19T18:50:26.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:25 smithi077 ceph-mon[29247]: Added host smithi077 2024-02-19T18:50:26.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:25 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:26.470 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-19T18:50:26.470 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.470+0000 7fb6c2f8a700 1 Processor -- start 2024-02-19T18:50:26.470 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.470+0000 7fb6c2f8a700 1 -- start start 2024-02-19T18:50:26.470 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.470+0000 7fb6c2f8a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:26.470 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.470+0000 7fb6c2f8a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc0790f0 con 0x7fb6bc078ce0 2024-02-19T18:50:26.470 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.470+0000 7fb6c0d26700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:26.470 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.470+0000 7fb6c0d26700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc0771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56486/0 (socket says 172.21.15.77:56486) 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.470+0000 7fb6c0d26700 1 -- 172.21.15.77:0/2341093183 learned_addr learned my addr 172.21.15.77:0/2341093183 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6c0d26700 1 -- 172.21.15.77:0/2341093183 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6bc077730 con 0x7fb6bc078ce0 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6c0d26700 1 --2- 172.21.15.77:0/2341093183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc0771f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00098e0 tx=0x7fb6b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae497407779e20a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6bb7fe700 1 -- 172.21.15.77:0/2341093183 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6b000b860 con 0x7fb6bc078ce0 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6bb7fe700 1 -- 172.21.15.77:0/2341093183 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb6b000be50 con 0x7fb6bc078ce0 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6bb7fe700 1 -- 172.21.15.77:0/2341093183 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6b0010380 con 0x7fb6bc078ce0 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/2341093183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 msgr2=0x7fb6bc0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6c2f8a700 1 --2- 172.21.15.77:0/2341093183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc0771f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6b00098e0 tx=0x7fb6b0009250 comp rx=0 tx=0).stop 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/2341093183 shutdown_connections 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6c2f8a700 1 --2- 172.21.15.77:0/2341093183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc0771f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/2341093183 >> 172.21.15.77:0/2341093183 conn(0x7fb6bc0fded0 msgr2=0x7fb6bc100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.471+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/2341093183 shutdown_connections 2024-02-19T18:50:26.471 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.472+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/2341093183 wait complete. 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.472+0000 7fb6c2f8a700 1 Processor -- start 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.472+0000 7fb6c2f8a700 1 -- start start 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.472+0000 7fb6c2f8a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc1137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.472+0000 7fb6c2f8a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6bc113d20 con 0x7fb6bc078ce0 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.472+0000 7fb6c0d26700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc1137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.472+0000 7fb6c0d26700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc1137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:56488/0 (socket says 172.21.15.77:56488) 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.472+0000 7fb6c0d26700 1 -- 172.21.15.77:0/683115067 learned_addr learned my addr 172.21.15.77:0/683115067 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.473+0000 7fb6c0d26700 1 -- 172.21.15.77:0/683115067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b00094f0 con 0x7fb6bc078ce0 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.473+0000 7fb6c0d26700 1 --2- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc1137e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000b440 tx=0x7fb6b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.473+0000 7fb6b97fa700 1 -- 172.21.15.77:0/683115067 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6b0010760 con 0x7fb6bc078ce0 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.473+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6bc113f20 con 0x7fb6bc078ce0 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.473+0000 7fb6b97fa700 1 -- 172.21.15.77:0/683115067 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb6b0010d50 con 0x7fb6bc078ce0 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.473+0000 7fb6b97fa700 1 -- 172.21.15.77:0/683115067 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6b001adc0 con 0x7fb6bc078ce0 2024-02-19T18:50:26.472 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.473+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6bc114bc0 con 0x7fb6bc078ce0 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.474+0000 7fb6b97fa700 1 -- 172.21.15.77:0/683115067 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb6b0019410 con 0x7fb6bc078ce0 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.474+0000 7fb6b97fa700 1 --2- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb6ac036770 0x7fb6ac038c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.474+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6bc05f5c0 con 0x7fb6bc078ce0 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.474+0000 7fb6b97fa700 1 -- 172.21.15.77:0/683115067 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb6b004a0f0 con 0x7fb6bc078ce0 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.474+0000 7fb6bbfff700 1 --2- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb6ac036770 0x7fb6ac038c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.474+0000 7fb6bbfff700 1 --2- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb6ac036770 0x7fb6ac038c20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009990 tx=0x7fb6a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.479+0000 7fb6b97fa700 1 -- 172.21.15.77:0/683115067 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb6b0022ab0 con 0x7fb6bc078ce0 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.681+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fb6bc1001f0 con 0x7fb6ac036770 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.686+0000 7fb6b97fa700 1 -- 172.21.15.77:0/683115067 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb6bc1001f0 con 0x7fb6ac036770 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.687+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb6ac036770 msgr2=0x7fb6ac038c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.687+0000 7fb6c2f8a700 1 --2- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb6ac036770 0x7fb6ac038c20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009990 tx=0x7fb6a8006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.687+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 msgr2=0x7fb6bc1137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.687+0000 7fb6c2f8a700 1 --2- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc1137e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6b000b440 tx=0x7fb6b0009250 comp rx=0 tx=0).stop 2024-02-19T18:50:26.473 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.688+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 shutdown_connections 2024-02-19T18:50:26.474 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.688+0000 7fb6c2f8a700 1 --2- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb6ac036770 0x7fb6ac038c20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:26.474 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.688+0000 7fb6c2f8a700 1 --2- 172.21.15.77:0/683115067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6bc078ce0 0x7fb6bc1137e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:26.474 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.688+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 >> 172.21.15.77:0/683115067 conn(0x7fb6bc0fded0 msgr2=0x7fb6bc0ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:26.474 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.688+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 shutdown_connections 2024-02-19T18:50:26.474 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:25.688+0000 7fb6c2f8a700 1 -- 172.21.15.77:0/683115067 wait complete. 2024-02-19T18:50:26.474 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying mgr service with default placement... 2024-02-19T18:50:27.024 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:26 smithi077 ceph-mon[29247]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:27.024 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:26 smithi077 ceph-mon[29247]: Saving service mon spec with placement count:5 2024-02-19T18:50:28.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:28 smithi077 ceph-mon[29247]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:28.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:28 smithi077 ceph-mon[29247]: Saving service mgr spec with placement count:2 2024-02-19T18:50:28.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:28 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.830+0000 7fdf781ae700 1 Processor -- start 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.830+0000 7fdf781ae700 1 -- start start 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.830+0000 7fdf781ae700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70104250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.830+0000 7fdf781ae700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf70104790 con 0x7fdf70103e40 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.830+0000 7fdf75f4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70104250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.830+0000 7fdf75f4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70104250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36206/0 (socket says 172.21.15.77:36206) 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.830+0000 7fdf75f4a700 1 -- 172.21.15.77:0/3627140833 learned_addr learned my addr 172.21.15.77:0/3627140833 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.830+0000 7fdf75f4a700 1 -- 172.21.15.77:0/3627140833 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf701048d0 con 0x7fdf70103e40 2024-02-19T18:50:28.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf75f4a700 1 --2- 172.21.15.77:0/3627140833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70104250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098e0 tx=0x7fdf6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e214cf18004afc server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf74f48700 1 -- 172.21.15.77:0/3627140833 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf6c00b860 con 0x7fdf70103e40 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf74f48700 1 -- 172.21.15.77:0/3627140833 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdf6c00be50 con 0x7fdf70103e40 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf74f48700 1 -- 172.21.15.77:0/3627140833 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf6c010380 con 0x7fdf70103e40 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf781ae700 1 -- 172.21.15.77:0/3627140833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 msgr2=0x7fdf70104250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf781ae700 1 --2- 172.21.15.77:0/3627140833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70104250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098e0 tx=0x7fdf6c009250 comp rx=0 tx=0).stop 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf781ae700 1 -- 172.21.15.77:0/3627140833 shutdown_connections 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf781ae700 1 --2- 172.21.15.77:0/3627140833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70104250 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf781ae700 1 -- 172.21.15.77:0/3627140833 >> 172.21.15.77:0/3627140833 conn(0x7fdf700fc580 msgr2=0x7fdf700fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.831+0000 7fdf781ae700 1 -- 172.21.15.77:0/3627140833 shutdown_connections 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf781ae700 1 -- 172.21.15.77:0/3627140833 wait complete. 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf781ae700 1 Processor -- start 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf781ae700 1 -- start start 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf781ae700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf781ae700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf70111d60 con 0x7fdf70103e40 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf75f4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:28.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf75f4a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70111820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36218/0 (socket says 172.21.15.77:36218) 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf75f4a700 1 -- 172.21.15.77:0/3074437193 learned_addr learned my addr 172.21.15.77:0/3074437193 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.832+0000 7fdf75f4a700 1 -- 172.21.15.77:0/3074437193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c0094f0 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.833+0000 7fdf75f4a700 1 --2- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70111820 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00b440 tx=0x7fdf6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.833+0000 7fdf667fc700 1 -- 172.21.15.77:0/3074437193 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf6c0107c0 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.833+0000 7fdf667fc700 1 -- 172.21.15.77:0/3074437193 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdf6c010db0 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.833+0000 7fdf667fc700 1 -- 172.21.15.77:0/3074437193 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf6c01ada0 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.833+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf70111f60 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.833+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf70112c00 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.833+0000 7fdf667fc700 1 -- 172.21.15.77:0/3074437193 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fdf6c010920 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.834+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7005f5c0 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.834+0000 7fdf667fc700 1 --2- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fdf5c036790 0x7fdf5c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.834+0000 7fdf667fc700 1 -- 172.21.15.77:0/3074437193 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdf6c04adc0 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.834+0000 7fdf75749700 1 --2- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fdf5c036790 0x7fdf5c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.834+0000 7fdf75749700 1 --2- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fdf5c036790 0x7fdf5c038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009990 tx=0x7fdf60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:26.838+0000 7fdf667fc700 1 -- 172.21.15.77:0/3074437193 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdf6c018530 con 0x7fdf70103e40 2024-02-19T18:50:28.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.037+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf70101480 con 0x7fdf5c036790 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.044+0000 7fdf667fc700 1 -- 172.21.15.77:0/3074437193 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fdf70101480 con 0x7fdf5c036790 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fdf5c036790 msgr2=0x7fdf5c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 --2- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fdf5c036790 0x7fdf5c038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdf60009990 tx=0x7fdf60006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 msgr2=0x7fdf70111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 --2- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70111820 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c00b440 tx=0x7fdf6c009250 comp rx=0 tx=0).stop 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 shutdown_connections 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 --2- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fdf5c036790 0x7fdf5c038c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 --2- 172.21.15.77:0/3074437193 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf70103e40 0x7fdf70111820 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 >> 172.21.15.77:0/3074437193 conn(0x7fdf700fc580 msgr2=0x7fdf700fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 shutdown_connections 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:27.046+0000 7fdf781ae700 1 -- 172.21.15.77:0/3074437193 wait complete. 2024-02-19T18:50:28.385 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying crash service with default placement... 2024-02-19T18:50:29.669 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:29 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:29.669 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:29 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:29.670 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:29 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-02-19T18:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.785+0000 7f99b8b6d700 1 Processor -- start 2024-02-19T18:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.785+0000 7f99b8b6d700 1 -- start start 2024-02-19T18:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b8b6d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4102040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:30.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b8b6d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4102580 con 0x7f99b4101c30 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4102040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36220/0 (socket says 172.21.15.77:36220) 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b259c700 1 -- 172.21.15.77:0/1463727222 learned_addr learned my addr 172.21.15.77:0/1463727222 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b259c700 1 -- 172.21.15.77:0/1463727222 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b41026c0 con 0x7f99b4101c30 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b259c700 1 --2- 172.21.15.77:0/1463727222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4102040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f99a40098e0 tx=0x7f99a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a09d7b2503a7686 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b159a700 1 -- 172.21.15.77:0/1463727222 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a400b860 con 0x7f99b4101c30 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b159a700 1 -- 172.21.15.77:0/1463727222 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99a400be90 con 0x7f99b4101c30 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.786+0000 7f99b159a700 1 -- 172.21.15.77:0/1463727222 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a40103c0 con 0x7f99b4101c30 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.787+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1463727222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 msgr2=0x7f99b4102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.787+0000 7f99b8b6d700 1 --2- 172.21.15.77:0/1463727222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4102040 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f99a40098e0 tx=0x7f99a4009250 comp rx=0 tx=0).stop 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.787+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1463727222 shutdown_connections 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.787+0000 7f99b8b6d700 1 --2- 172.21.15.77:0/1463727222 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4102040 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.787+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1463727222 >> 172.21.15.77:0/1463727222 conn(0x7f99b40fc580 msgr2=0x7f99b40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:30.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.787+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1463727222 shutdown_connections 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.787+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1463727222 wait complete. 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b8b6d700 1 Processor -- start 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b8b6d700 1 -- start start 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b8b6d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4110750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b8b6d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b4110c90 con 0x7f99b4101c30 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4110750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4110750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36234/0 (socket says 172.21.15.77:36234) 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b259c700 1 -- 172.21.15.77:0/1446846242 learned_addr learned my addr 172.21.15.77:0/1446846242 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b259c700 1 -- 172.21.15.77:0/1446846242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a40094f0 con 0x7f99b4101c30 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.788+0000 7f99b259c700 1 --2- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4110750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f99a4006b00 tx=0x7f99a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.789+0000 7f99a2ffd700 1 -- 172.21.15.77:0/1446846242 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a4010790 con 0x7f99b4101c30 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.789+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b4110e90 con 0x7f99b4101c30 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.789+0000 7f99a2ffd700 1 -- 172.21.15.77:0/1446846242 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f99a4010dc0 con 0x7f99b4101c30 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.789+0000 7f99a2ffd700 1 -- 172.21.15.77:0/1446846242 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a401ae70 con 0x7f99b4101c30 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.789+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b4111b30 con 0x7f99b4101c30 2024-02-19T18:50:30.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.789+0000 7f99a2ffd700 1 -- 172.21.15.77:0/1446846242 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f99a4022030 con 0x7f99b4101c30 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.790+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b405f5c0 con 0x7f99b4101c30 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.790+0000 7f99a2ffd700 1 --2- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f999c036790 0x7f999c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.790+0000 7f99a2ffd700 1 -- 172.21.15.77:0/1446846242 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f99a404c760 con 0x7f99b4101c30 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.790+0000 7f99b1d9b700 1 --2- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f999c036790 0x7f999c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.790+0000 7f99b1d9b700 1 --2- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f999c036790 0x7f999c038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009990 tx=0x7f99a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:28.795+0000 7f99a2ffd700 1 -- 172.21.15.77:0/1446846242 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f99a402f440 con 0x7f99b4101c30 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.006+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f99b405f5c0 con 0x7f999c036790 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.011+0000 7f99a2ffd700 1 -- 172.21.15.77:0/1446846242 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f99b405f5c0 con 0x7f999c036790 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f999c036790 msgr2=0x7f999c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:30.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 --2- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f999c036790 0x7f999c038c40 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009990 tx=0x7f99a8006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 msgr2=0x7f99b4110750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 --2- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4110750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f99a4006b00 tx=0x7f99a4009250 comp rx=0 tx=0).stop 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 shutdown_connections 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 --2- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f999c036790 0x7f999c038c40 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 --2- 172.21.15.77:0/1446846242 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f99b4101c30 0x7f99b4110750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 >> 172.21.15.77:0/1446846242 conn(0x7f99b40fc580 msgr2=0x7f99b40fe830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 shutdown_connections 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:29.013+0000 7f99b8b6d700 1 -- 172.21.15.77:0/1446846242 wait complete. 2024-02-19T18:50:30.221 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying prometheus service with default placement... 2024-02-19T18:50:30.889 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:30 smithi077 ceph-mon[29247]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:30.889 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:30 smithi077 ceph-mon[29247]: Saving service crash spec with placement * 2024-02-19T18:50:30.890 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:30 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:31.482 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.647+0000 7f17adcec700 1 Processor -- start 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.647+0000 7f17adcec700 1 -- start start 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.647+0000 7f17adcec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a8103020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.647+0000 7f17adcec700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a8103560 con 0x7f17a8102c10 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.648+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a8103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.648+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a8103020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36246/0 (socket says 172.21.15.77:36246) 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.648+0000 7f17a77fe700 1 -- 172.21.15.77:0/1683084989 learned_addr learned my addr 172.21.15.77:0/1683084989 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.648+0000 7f17a77fe700 1 -- 172.21.15.77:0/1683084989 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a81036a0 con 0x7f17a8102c10 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.648+0000 7f17a77fe700 1 --2- 172.21.15.77:0/1683084989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a8103020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009e50 tx=0x7f17a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e6be81dc31ace0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.649+0000 7f17a67fc700 1 -- 172.21.15.77:0/1683084989 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17a00092e0 con 0x7f17a8102c10 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.649+0000 7f17a67fc700 1 -- 172.21.15.77:0/1683084989 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17a0012430 con 0x7f17a8102c10 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.649+0000 7f17a67fc700 1 -- 172.21.15.77:0/1683084989 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17a00075a0 con 0x7f17a8102c10 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.649+0000 7f17adcec700 1 -- 172.21.15.77:0/1683084989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 msgr2=0x7f17a8103020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.649+0000 7f17adcec700 1 --2- 172.21.15.77:0/1683084989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a8103020 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009e50 tx=0x7f17a0009840 comp rx=0 tx=0).stop 2024-02-19T18:50:31.483 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.649+0000 7f17adcec700 1 -- 172.21.15.77:0/1683084989 shutdown_connections 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.650+0000 7f17adcec700 1 --2- 172.21.15.77:0/1683084989 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a8103020 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.650+0000 7f17adcec700 1 -- 172.21.15.77:0/1683084989 >> 172.21.15.77:0/1683084989 conn(0x7f17a80fc580 msgr2=0x7f17a80fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.650+0000 7f17adcec700 1 -- 172.21.15.77:0/1683084989 shutdown_connections 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.650+0000 7f17adcec700 1 -- 172.21.15.77:0/1683084989 wait complete. 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.650+0000 7f17adcec700 1 Processor -- start 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.650+0000 7f17adcec700 1 -- start start 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.651+0000 7f17adcec700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a810f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.651+0000 7f17adcec700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17a810fc00 con 0x7f17a8102c10 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.651+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a810f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.651+0000 7f17a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a810f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36258/0 (socket says 172.21.15.77:36258) 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.651+0000 7f17a77fe700 1 -- 172.21.15.77:0/227057102 learned_addr learned my addr 172.21.15.77:0/227057102 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.651+0000 7f17a77fe700 1 -- 172.21.15.77:0/227057102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a0009ae0 con 0x7f17a8102c10 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.651+0000 7f17a77fe700 1 --2- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a810f6c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f17a00127d0 tx=0x7f17a0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.652+0000 7f178ffff700 1 -- 172.21.15.77:0/227057102 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17a00092e0 con 0x7f17a8102c10 2024-02-19T18:50:31.484 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.652+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17a810fe00 con 0x7f17a8102c10 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.652+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17a8110290 con 0x7f17a8102c10 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.652+0000 7f178ffff700 1 -- 172.21.15.77:0/227057102 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17a0007d30 con 0x7f17a8102c10 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.652+0000 7f178ffff700 1 -- 172.21.15.77:0/227057102 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17a0025910 con 0x7f17a8102c10 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.653+0000 7f178ffff700 1 -- 172.21.15.77:0/227057102 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f17a0019050 con 0x7f17a8102c10 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.653+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17a804bed0 con 0x7f17a8102c10 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.653+0000 7f178ffff700 1 --2- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f17900367e0 0x7f1790038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.653+0000 7f178ffff700 1 -- 172.21.15.77:0/227057102 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f17a004a220 con 0x7f17a8102c10 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.653+0000 7f17a6ffd700 1 --2- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f17900367e0 0x7f1790038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.654+0000 7f17a6ffd700 1 --2- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f17900367e0 0x7f1790038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f179c009990 tx=0x7f179c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.658+0000 7f178ffff700 1 -- 172.21.15.77:0/227057102 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f17a002d760 con 0x7f17a8102c10 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.874+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f17a8113da0 con 0x7f17900367e0 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.878+0000 7f178ffff700 1 -- 172.21.15.77:0/227057102 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f17a8113da0 con 0x7f17900367e0 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f17900367e0 msgr2=0x7f1790038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 --2- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f17900367e0 0x7f1790038c90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f179c009990 tx=0x7f179c006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:31.485 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 msgr2=0x7f17a810f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 --2- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a810f6c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f17a00127d0 tx=0x7f17a0009840 comp rx=0 tx=0).stop 2024-02-19T18:50:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 shutdown_connections 2024-02-19T18:50:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 --2- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f17900367e0 0x7f1790038c90 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 --2- 172.21.15.77:0/227057102 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17a8102c10 0x7f17a810f6c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 >> 172.21.15.77:0/227057102 conn(0x7f17a80fc580 msgr2=0x7f17a80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 shutdown_connections 2024-02-19T18:50:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:30.880+0000 7f17adcec700 1 -- 172.21.15.77:0/227057102 wait complete. 2024-02-19T18:50:31.486 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying grafana service with default placement... 2024-02-19T18:50:31.802 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:31 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:31.803 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:31 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:32.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:32 smithi077 ceph-mon[29247]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:32.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:32 smithi077 ceph-mon[29247]: Saving service prometheus spec with placement count:1 2024-02-19T18:50:32.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:32 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:33.528 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-02-19T18:50:33.528 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.884+0000 7fe361b82700 1 Processor -- start 2024-02-19T18:50:33.528 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.884+0000 7fe361b82700 1 -- start start 2024-02-19T18:50:33.528 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.884+0000 7fe361b82700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c104270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:33.528 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.884+0000 7fe361b82700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c1047b0 con 0x7fe35c103e60 2024-02-19T18:50:33.528 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.885+0000 7fe35b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c104270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:33.528 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.885+0000 7fe35b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c104270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36262/0 (socket says 172.21.15.77:36262) 2024-02-19T18:50:33.528 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.885+0000 7fe35b7fe700 1 -- 172.21.15.77:0/2646630381 learned_addr learned my addr 172.21.15.77:0/2646630381 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.885+0000 7fe35b7fe700 1 -- 172.21.15.77:0/2646630381 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c1048f0 con 0x7fe35c103e60 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.885+0000 7fe35b7fe700 1 --2- 172.21.15.77:0/2646630381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c104270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe3500098e0 tx=0x7fe350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1dcab1435576300 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.885+0000 7fe35a7fc700 1 -- 172.21.15.77:0/2646630381 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe35000b860 con 0x7fe35c103e60 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.885+0000 7fe35a7fc700 1 -- 172.21.15.77:0/2646630381 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe35000be90 con 0x7fe35c103e60 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.885+0000 7fe35a7fc700 1 -- 172.21.15.77:0/2646630381 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe3500103c0 con 0x7fe35c103e60 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.886+0000 7fe361b82700 1 -- 172.21.15.77:0/2646630381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 msgr2=0x7fe35c104270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.886+0000 7fe361b82700 1 --2- 172.21.15.77:0/2646630381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c104270 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe3500098e0 tx=0x7fe350009250 comp rx=0 tx=0).stop 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.886+0000 7fe361b82700 1 -- 172.21.15.77:0/2646630381 shutdown_connections 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.886+0000 7fe361b82700 1 --2- 172.21.15.77:0/2646630381 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c104270 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.886+0000 7fe361b82700 1 -- 172.21.15.77:0/2646630381 >> 172.21.15.77:0/2646630381 conn(0x7fe35c0fc580 msgr2=0x7fe35c0fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.886+0000 7fe361b82700 1 -- 172.21.15.77:0/2646630381 shutdown_connections 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.886+0000 7fe361b82700 1 -- 172.21.15.77:0/2646630381 wait complete. 2024-02-19T18:50:33.529 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.886+0000 7fe361b82700 1 Processor -- start 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe361b82700 1 -- start start 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe361b82700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c111830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe361b82700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe35c111d70 con 0x7fe35c103e60 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe35b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c111830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe35b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c111830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36274/0 (socket says 172.21.15.77:36274) 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe35b7fe700 1 -- 172.21.15.77:0/2551627264 learned_addr learned my addr 172.21.15.77:0/2551627264 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe35b7fe700 1 -- 172.21.15.77:0/2551627264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3500094f0 con 0x7fe35c103e60 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe35b7fe700 1 --2- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c111830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe350011650 tx=0x7fe350009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe343fff700 1 -- 172.21.15.77:0/2551627264 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe350010890 con 0x7fe35c103e60 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe343fff700 1 -- 172.21.15.77:0/2551627264 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe350010ec0 con 0x7fe35c103e60 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe35c111f70 con 0x7fe35c103e60 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe343fff700 1 -- 172.21.15.77:0/2551627264 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe35001a430 con 0x7fe35c103e60 2024-02-19T18:50:33.530 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.887+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe35c112c10 con 0x7fe35c103e60 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.888+0000 7fe343fff700 1 -- 172.21.15.77:0/2551627264 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe35001a6b0 con 0x7fe35c103e60 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.888+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe35c05f5c0 con 0x7fe35c103e60 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.889+0000 7fe343fff700 1 --2- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe344036790 0x7fe344038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.889+0000 7fe343fff700 1 -- 172.21.15.77:0/2551627264 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe35004a670 con 0x7fe35c103e60 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.889+0000 7fe35affd700 1 --2- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe344036790 0x7fe344038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.889+0000 7fe35affd700 1 --2- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe344036790 0x7fe344038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009990 tx=0x7fe34c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:31.893+0000 7fe343fff700 1 -- 172.21.15.77:0/2551627264 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe350018db0 con 0x7fe35c103e60 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.111+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fe35c101410 con 0x7fe344036790 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.116+0000 7fe343fff700 1 -- 172.21.15.77:0/2551627264 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe35c101410 con 0x7fe344036790 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe344036790 msgr2=0x7fe344038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 --2- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe344036790 0x7fe344038c40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe34c009990 tx=0x7fe34c006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 msgr2=0x7fe35c111830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 --2- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c111830 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe350011650 tx=0x7fe350009250 comp rx=0 tx=0).stop 2024-02-19T18:50:33.532 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 shutdown_connections 2024-02-19T18:50:33.532 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 --2- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe344036790 0x7fe344038c40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:33.532 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 --2- 172.21.15.77:0/2551627264 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe35c103e60 0x7fe35c111830 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:33.532 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 >> 172.21.15.77:0/2551627264 conn(0x7fe35c0fc580 msgr2=0x7fe35c0fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:33.532 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 shutdown_connections 2024-02-19T18:50:33.532 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:32.118+0000 7fe361b82700 1 -- 172.21.15.77:0/2551627264 wait complete. 2024-02-19T18:50:33.532 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying node-exporter service with default placement... 2024-02-19T18:50:34.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:33 smithi077 ceph-mon[29247]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:34.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:33 smithi077 ceph-mon[29247]: Saving service grafana spec with placement count:1 2024-02-19T18:50:34.683 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.912+0000 7f6579962700 1 Processor -- start 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.912+0000 7f6579962700 1 -- start start 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.912+0000 7f6579962700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f6574105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.912+0000 7f6579962700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6574105960 con 0x7f6574076b80 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.912+0000 7f6572ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f6574105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.912+0000 7f6572ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f6574105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36284/0 (socket says 172.21.15.77:36284) 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.912+0000 7f6572ffd700 1 -- 172.21.15.77:0/4034316314 learned_addr learned my addr 172.21.15.77:0/4034316314 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6572ffd700 1 -- 172.21.15.77:0/4034316314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6574105aa0 con 0x7f6574076b80 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6572ffd700 1 --2- 172.21.15.77:0/4034316314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f6574105420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f65640098e0 tx=0x7f6564009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e3ca9cba86d8e2c server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6571ffb700 1 -- 172.21.15.77:0/4034316314 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f656400b860 con 0x7f6574076b80 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6571ffb700 1 -- 172.21.15.77:0/4034316314 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f656400be90 con 0x7f6574076b80 2024-02-19T18:50:34.684 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6571ffb700 1 -- 172.21.15.77:0/4034316314 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f65640103c0 con 0x7f6574076b80 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6579962700 1 -- 172.21.15.77:0/4034316314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 msgr2=0x7f6574105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6579962700 1 --2- 172.21.15.77:0/4034316314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f6574105420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f65640098e0 tx=0x7f6564009250 comp rx=0 tx=0).stop 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6579962700 1 -- 172.21.15.77:0/4034316314 shutdown_connections 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6579962700 1 --2- 172.21.15.77:0/4034316314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f6574105420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6579962700 1 -- 172.21.15.77:0/4034316314 >> 172.21.15.77:0/4034316314 conn(0x7f65740fcdd0 msgr2=0x7f65740ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.913+0000 7f6579962700 1 -- 172.21.15.77:0/4034316314 shutdown_connections 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.914+0000 7f6579962700 1 -- 172.21.15.77:0/4034316314 wait complete. 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.914+0000 7f6579962700 1 Processor -- start 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.914+0000 7f6579962700 1 -- start start 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.914+0000 7f6579962700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f65741115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.914+0000 7f6579962700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6574111af0 con 0x7f6574076b80 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.914+0000 7f6572ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f65741115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.914+0000 7f6572ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f65741115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36288/0 (socket says 172.21.15.77:36288) 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.914+0000 7f6572ffd700 1 -- 172.21.15.77:0/2548985113 learned_addr learned my addr 172.21.15.77:0/2548985113 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:34.685 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.915+0000 7f6572ffd700 1 -- 172.21.15.77:0/2548985113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65640094f0 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.915+0000 7f6572ffd700 1 --2- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f65741115b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f65640090b0 tx=0x7f6564009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.915+0000 7f655b7fe700 1 -- 172.21.15.77:0/2548985113 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6564010890 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.915+0000 7f655b7fe700 1 -- 172.21.15.77:0/2548985113 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6564010ec0 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.915+0000 7f655b7fe700 1 -- 172.21.15.77:0/2548985113 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6564024e70 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.915+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6574111cf0 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.915+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6574112990 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.916+0000 7f655b7fe700 1 -- 172.21.15.77:0/2548985113 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f656401a550 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.916+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657405f5c0 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.916+0000 7f655b7fe700 1 --2- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f655c036790 0x7f655c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.916+0000 7f655b7fe700 1 -- 172.21.15.77:0/2548985113 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f656404aa20 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.916+0000 7f65727fc700 1 --2- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f655c036790 0x7f655c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.916+0000 7f65727fc700 1 --2- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f655c036790 0x7f655c038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6568009990 tx=0x7f6568006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:33.921+0000 7f655b7fe700 1 -- 172.21.15.77:0/2548985113 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6564018940 con 0x7f6574076b80 2024-02-19T18:50:34.686 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.114+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f65740ff090 con 0x7f655c036790 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.118+0000 7f655b7fe700 1 -- 172.21.15.77:0/2548985113 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f65740ff090 con 0x7f655c036790 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.120+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f655c036790 msgr2=0x7f655c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.120+0000 7f6579962700 1 --2- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f655c036790 0x7f655c038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6568009990 tx=0x7f6568006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.120+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 msgr2=0x7f65741115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.120+0000 7f6579962700 1 --2- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f65741115b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f65640090b0 tx=0x7f6564009250 comp rx=0 tx=0).stop 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.120+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 shutdown_connections 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.120+0000 7f6579962700 1 --2- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f655c036790 0x7f655c038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.120+0000 7f6579962700 1 --2- 172.21.15.77:0/2548985113 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6574076b80 0x7f65741115b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.120+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 >> 172.21.15.77:0/2548985113 conn(0x7f65740fcdd0 msgr2=0x7f65740fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.121+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 shutdown_connections 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:34.121+0000 7f6579962700 1 -- 172.21.15.77:0/2548985113 wait complete. 2024-02-19T18:50:34.687 INFO:teuthology.orchestra.run.smithi077.stdout:Deploying alertmanager service with default placement... 2024-02-19T18:50:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:35 smithi077 ceph-mon[29247]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:35 smithi077 ceph-mon[29247]: Saving service node-exporter spec with placement * 2024-02-19T18:50:35.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:35 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:35.935 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-02-19T18:50:35.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.108+0000 7fb58453a700 1 Processor -- start 2024-02-19T18:50:35.945 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.108+0000 7fb58453a700 1 -- start start 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb58453a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c0771f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb58453a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0790f0 con 0x7fb57c078ce0 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb5822d6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c0771f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb5822d6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c0771f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36302/0 (socket says 172.21.15.77:36302) 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb5822d6700 1 -- 172.21.15.77:0/2947295675 learned_addr learned my addr 172.21.15.77:0/2947295675 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb5822d6700 1 -- 172.21.15.77:0/2947295675 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c077730 con 0x7fb57c078ce0 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb5822d6700 1 --2- 172.21.15.77:0/2947295675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c0771f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0098e0 tx=0x7fb56c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bacb347bfabc3fb server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb5812d4700 1 -- 172.21.15.77:0/2947295675 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb56c00b860 con 0x7fb57c078ce0 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb5812d4700 1 -- 172.21.15.77:0/2947295675 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb56c00be90 con 0x7fb57c078ce0 2024-02-19T18:50:35.946 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.109+0000 7fb5812d4700 1 -- 172.21.15.77:0/2947295675 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb56c0103c0 con 0x7fb57c078ce0 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.110+0000 7fb58453a700 1 -- 172.21.15.77:0/2947295675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 msgr2=0x7fb57c0771f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.110+0000 7fb58453a700 1 --2- 172.21.15.77:0/2947295675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c0771f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0098e0 tx=0x7fb56c009250 comp rx=0 tx=0).stop 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.110+0000 7fb58453a700 1 -- 172.21.15.77:0/2947295675 shutdown_connections 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.110+0000 7fb58453a700 1 --2- 172.21.15.77:0/2947295675 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c0771f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.110+0000 7fb58453a700 1 -- 172.21.15.77:0/2947295675 >> 172.21.15.77:0/2947295675 conn(0x7fb57c0fded0 msgr2=0x7fb57c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.110+0000 7fb58453a700 1 -- 172.21.15.77:0/2947295675 shutdown_connections 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.110+0000 7fb58453a700 1 -- 172.21.15.77:0/2947295675 wait complete. 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb58453a700 1 Processor -- start 2024-02-19T18:50:35.947 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb58453a700 1 -- start start 2024-02-19T18:50:35.948 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb58453a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c113750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:35.948 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb58453a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c113c90 con 0x7fb57c078ce0 2024-02-19T18:50:35.948 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb5822d6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c113750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:35.948 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb5822d6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c113750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36314/0 (socket says 172.21.15.77:36314) 2024-02-19T18:50:35.948 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb5822d6700 1 -- 172.21.15.77:0/1719585503 learned_addr learned my addr 172.21.15.77:0/1719585503 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:35.948 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb5822d6700 1 -- 172.21.15.77:0/1719585503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0094f0 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.111+0000 7fb5822d6700 1 --2- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c113750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0090b0 tx=0x7fb56c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.112+0000 7fb572ffd700 1 -- 172.21.15.77:0/1719585503 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb56c0118f0 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.112+0000 7fb572ffd700 1 -- 172.21.15.77:0/1719585503 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb56c010d10 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.112+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c113e90 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.112+0000 7fb572ffd700 1 -- 172.21.15.77:0/1719585503 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb56c024e70 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.112+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c114b30 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.112+0000 7fb572ffd700 1 -- 172.21.15.77:0/1719585503 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb56c01a500 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.113+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c05f5c0 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.113+0000 7fb572ffd700 1 --2- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb568036790 0x7fb568038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.113+0000 7fb572ffd700 1 -- 172.21.15.77:0/1719585503 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb56c04a690 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.113+0000 7fb581ad5700 1 --2- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb568036790 0x7fb568038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.113+0000 7fb581ad5700 1 --2- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb568036790 0x7fb568038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb578009990 tx=0x7fb578006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.117+0000 7fb572ffd700 1 -- 172.21.15.77:0/1719585503 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb56c01a7b0 con 0x7fb57c078ce0 2024-02-19T18:50:35.949 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.308+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fb57c1001b0 con 0x7fb568036790 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.313+0000 7fb572ffd700 1 -- 172.21.15.77:0/1719585503 <== mgr.14120 v2:172.21.15.77:6800/1028404574 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fb57c1001b0 con 0x7fb568036790 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb568036790 msgr2=0x7fb568038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 --2- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb568036790 0x7fb568038c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb578009990 tx=0x7fb578006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 msgr2=0x7fb57c113750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 --2- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c113750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0090b0 tx=0x7fb56c009250 comp rx=0 tx=0).stop 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 shutdown_connections 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 --2- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb568036790 0x7fb568038c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 --2- 172.21.15.77:0/1719585503 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb57c078ce0 0x7fb57c113750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 >> 172.21.15.77:0/1719585503 conn(0x7fb57c0fded0 msgr2=0x7fb57c0ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 shutdown_connections 2024-02-19T18:50:35.950 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:35.315+0000 7fb58453a700 1 -- 172.21.15.77:0/1719585503 wait complete. 2024-02-19T18:50:36.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:36 smithi077 ceph-mon[29247]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:36.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:36 smithi077 ceph-mon[29247]: Saving service alertmanager spec with placement count:1 2024-02-19T18:50:36.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:36 smithi077 ceph-mon[29247]: from='mgr.14120 172.21.15.77:0/3658709087' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:36.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:36 smithi077 ceph-mon[29247]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:50:37.062 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.311+0000 7fb67c8c8700 1 Processor -- start 2024-02-19T18:50:37.062 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.311+0000 7fb67c8c8700 1 -- start start 2024-02-19T18:50:37.062 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.311+0000 7fb67c8c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb678102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.311+0000 7fb67c8c8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6781030c0 con 0x7fb678102770 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.311+0000 7fb67659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb678102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.311+0000 7fb67659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb678102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36316/0 (socket says 172.21.15.77:36316) 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.311+0000 7fb67659c700 1 -- 172.21.15.77:0/3323153046 learned_addr learned my addr 172.21.15.77:0/3323153046 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.311+0000 7fb67659c700 1 -- 172.21.15.77:0/3323153046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb678103200 con 0x7fb678102770 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.312+0000 7fb67659c700 1 --2- 172.21.15.77:0/3323153046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb678102b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb6680098e0 tx=0x7fb668009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388caf86567d4daf server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67559a700 1 -- 172.21.15.77:0/3323153046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb66800b860 con 0x7fb678102770 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67559a700 1 -- 172.21.15.77:0/3323153046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb66800be90 con 0x7fb678102770 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67559a700 1 -- 172.21.15.77:0/3323153046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6680103c0 con 0x7fb678102770 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3323153046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 msgr2=0x7fb678102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67c8c8700 1 --2- 172.21.15.77:0/3323153046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb678102b80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb6680098e0 tx=0x7fb668009250 comp rx=0 tx=0).stop 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3323153046 shutdown_connections 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67c8c8700 1 --2- 172.21.15.77:0/3323153046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb678102b80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3323153046 >> 172.21.15.77:0/3323153046 conn(0x7fb6780fded0 msgr2=0x7fb678100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:37.063 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.314+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3323153046 shutdown_connections 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.315+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3323153046 wait complete. 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.315+0000 7fb67c8c8700 1 Processor -- start 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.315+0000 7fb67c8c8700 1 -- start start 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.315+0000 7fb67c8c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb6781115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.315+0000 7fb67c8c8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb678111af0 con 0x7fb678102770 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.315+0000 7fb67659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb6781115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.315+0000 7fb67659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb6781115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:36320/0 (socket says 172.21.15.77:36320) 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.315+0000 7fb67659c700 1 -- 172.21.15.77:0/3253492340 learned_addr learned my addr 172.21.15.77:0/3253492340 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.316+0000 7fb67659c700 1 -- 172.21.15.77:0/3253492340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6680094f0 con 0x7fb678102770 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.316+0000 7fb67659c700 1 --2- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb6781115b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb6680090b0 tx=0x7fb668009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.316+0000 7fb666ffd700 1 -- 172.21.15.77:0/3253492340 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb668010840 con 0x7fb678102770 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.316+0000 7fb666ffd700 1 -- 172.21.15.77:0/3253492340 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb668010e70 con 0x7fb678102770 2024-02-19T18:50:37.064 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.316+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb678111cf0 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.316+0000 7fb666ffd700 1 -- 172.21.15.77:0/3253492340 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb668024e40 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.316+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb678112990 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.317+0000 7fb666ffd700 1 -- 172.21.15.77:0/3253492340 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb66801a570 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.317+0000 7fb666ffd700 1 --2- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb660036740 0x7fb660038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.317+0000 7fb666ffd700 1 -- 172.21.15.77:0/3253492340 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb66804a820 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.317+0000 7fb675d9b700 1 --2- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb660036740 0x7fb660038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.317+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67805f5c0 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.317+0000 7fb675d9b700 1 --2- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb660036740 0x7fb660038bf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009990 tx=0x7fb66c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.322+0000 7fb666ffd700 1 -- 172.21.15.77:0/3253492340 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb668018940 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.505+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb67804bed0 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.509+0000 7fb666ffd700 1 -- 172.21.15.77:0/3253492340 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb668048020 con 0x7fb678102770 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb660036740 msgr2=0x7fb660038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 --2- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb660036740 0x7fb660038bf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009990 tx=0x7fb66c006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 msgr2=0x7fb6781115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:37.065 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 --2- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb6781115b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb6680090b0 tx=0x7fb668009250 comp rx=0 tx=0).stop 2024-02-19T18:50:37.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 shutdown_connections 2024-02-19T18:50:37.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 --2- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fb660036740 0x7fb660038bf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:37.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 --2- 172.21.15.77:0/3253492340 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb678102770 0x7fb6781115b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:37.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 >> 172.21.15.77:0/3253492340 conn(0x7fb6780fded0 msgr2=0x7fb6780ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:37.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 shutdown_connections 2024-02-19T18:50:37.066 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:36.511+0000 7fb67c8c8700 1 -- 172.21.15.77:0/3253492340 wait complete. 2024-02-19T18:50:37.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:37 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3253492340' entity='client.admin' 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.432+0000 7f4aba879700 1 Processor -- start 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.432+0000 7f4aba879700 1 -- start start 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.432+0000 7f4aba879700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.432+0000 7f4aba879700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4101fc0 con 0x7f4ab4101670 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.432+0000 7f4ab3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.432+0000 7f4ab3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40360/0 (socket says 172.21.15.77:40360) 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.432+0000 7f4ab3fff700 1 -- 172.21.15.77:0/1109513875 learned_addr learned my addr 172.21.15.77:0/1109513875 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.432+0000 7f4ab3fff700 1 -- 172.21.15.77:0/1109513875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4102100 con 0x7f4ab4101670 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4ab3fff700 1 --2- 172.21.15.77:0/1109513875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4101a80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40098e0 tx=0x7f4aa4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af2f6e8e694ba194 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4ab2ffd700 1 -- 172.21.15.77:0/1109513875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa400b860 con 0x7f4ab4101670 2024-02-19T18:50:38.228 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4ab2ffd700 1 -- 172.21.15.77:0/1109513875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4aa400be90 con 0x7f4ab4101670 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4ab2ffd700 1 -- 172.21.15.77:0/1109513875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa40103c0 con 0x7f4ab4101670 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4aba879700 1 -- 172.21.15.77:0/1109513875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 msgr2=0x7f4ab4101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4aba879700 1 --2- 172.21.15.77:0/1109513875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4101a80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40098e0 tx=0x7f4aa4009250 comp rx=0 tx=0).stop 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4aba879700 1 -- 172.21.15.77:0/1109513875 shutdown_connections 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4aba879700 1 --2- 172.21.15.77:0/1109513875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4101a80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4aba879700 1 -- 172.21.15.77:0/1109513875 >> 172.21.15.77:0/1109513875 conn(0x7f4ab40fcdd0 msgr2=0x7f4ab40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4aba879700 1 -- 172.21.15.77:0/1109513875 shutdown_connections 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.433+0000 7f4aba879700 1 -- 172.21.15.77:0/1109513875 wait complete. 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.434+0000 7f4aba879700 1 Processor -- start 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.434+0000 7f4aba879700 1 -- start start 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.434+0000 7f4aba879700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4111890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.434+0000 7f4aba879700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4111dd0 con 0x7f4ab4101670 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.434+0000 7f4ab3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4111890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.434+0000 7f4ab3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4111890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40364/0 (socket says 172.21.15.77:40364) 2024-02-19T18:50:38.229 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.434+0000 7f4ab3fff700 1 -- 172.21.15.77:0/1476558815 learned_addr learned my addr 172.21.15.77:0/1476558815 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.435+0000 7f4ab3fff700 1 -- 172.21.15.77:0/1476558815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa40094f0 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.435+0000 7f4ab3fff700 1 --2- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4111890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4011650 tx=0x7f4aa4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.435+0000 7f4ab0ff9700 1 -- 172.21.15.77:0/1476558815 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa4010890 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.435+0000 7f4ab0ff9700 1 -- 172.21.15.77:0/1476558815 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4aa4010ec0 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.435+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab4111fd0 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.435+0000 7f4ab0ff9700 1 -- 172.21.15.77:0/1476558815 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4aa401a430 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.435+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab4112c70 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.436+0000 7f4ab0ff9700 1 -- 172.21.15.77:0/1476558815 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f4aa401a6b0 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.436+0000 7f4ab0ff9700 1 --2- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f4a9c036790 0x7f4a9c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.436+0000 7f4ab0ff9700 1 -- 172.21.15.77:0/1476558815 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4aa404b590 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.436+0000 7f4ab37fe700 1 --2- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f4a9c036790 0x7f4a9c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.436+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab4112460 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.436+0000 7f4ab37fe700 1 --2- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f4a9c036790 0x7f4a9c038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009990 tx=0x7f4aa8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.441+0000 7f4ab0ff9700 1 -- 172.21.15.77:0/1476558815 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4aa402d3f0 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.624+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f4ab4064b00 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.628+0000 7f4ab0ff9700 1 -- 172.21.15.77:0/1476558815 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f4aa4018350 con 0x7f4ab4101670 2024-02-19T18:50:38.230 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.630+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f4a9c036790 msgr2=0x7f4a9c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.630+0000 7f4aba879700 1 --2- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f4a9c036790 0x7f4a9c038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009990 tx=0x7f4aa8006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.630+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 msgr2=0x7f4ab4111890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.630+0000 7f4aba879700 1 --2- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4111890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4011650 tx=0x7f4aa4009250 comp rx=0 tx=0).stop 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.630+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 shutdown_connections 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.630+0000 7f4aba879700 1 --2- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f4a9c036790 0x7f4a9c038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.630+0000 7f4aba879700 1 --2- 172.21.15.77:0/1476558815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4ab4101670 0x7f4ab4111890 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.630+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 >> 172.21.15.77:0/1476558815 conn(0x7f4ab40fcdd0 msgr2=0x7f4ab40fdd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.631+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 shutdown_connections 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:37.631+0000 7f4aba879700 1 -- 172.21.15.77:0/1476558815 wait complete. 2024-02-19T18:50:38.231 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling the dashboard module... 2024-02-19T18:50:38.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:38 smithi077 ceph-mon[29247]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:50:38.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:38 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1476558815' entity='client.admin' 2024-02-19T18:50:39.951 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:39 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3646687046' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-19T18:50:40.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.613+0000 7fe151b5d700 1 Processor -- start 2024-02-19T18:50:40.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.613+0000 7fe151b5d700 1 -- start start 2024-02-19T18:50:40.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.613+0000 7fe151b5d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:40.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.613+0000 7fe151b5d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c101fc0 con 0x7fe14c101670 2024-02-19T18:50:40.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.613+0000 7fe14b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:40.217 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.613+0000 7fe14b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40368/0 (socket says 172.21.15.77:40368) 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.613+0000 7fe14b7fe700 1 -- 172.21.15.77:0/2750982721 learned_addr learned my addr 172.21.15.77:0/2750982721 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe14b7fe700 1 -- 172.21.15.77:0/2750982721 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe14c102100 con 0x7fe14c101670 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe14b7fe700 1 --2- 172.21.15.77:0/2750982721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c101a80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe13c0098e0 tx=0x7fe13c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16faa5c4e748132b server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe14a7fc700 1 -- 172.21.15.77:0/2750982721 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe13c00b860 con 0x7fe14c101670 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe14a7fc700 1 -- 172.21.15.77:0/2750982721 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe13c00be90 con 0x7fe14c101670 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe14a7fc700 1 -- 172.21.15.77:0/2750982721 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe13c0103c0 con 0x7fe14c101670 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe151b5d700 1 -- 172.21.15.77:0/2750982721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 msgr2=0x7fe14c101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe151b5d700 1 --2- 172.21.15.77:0/2750982721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c101a80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe13c0098e0 tx=0x7fe13c009250 comp rx=0 tx=0).stop 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe151b5d700 1 -- 172.21.15.77:0/2750982721 shutdown_connections 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe151b5d700 1 --2- 172.21.15.77:0/2750982721 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c101a80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.614+0000 7fe151b5d700 1 -- 172.21.15.77:0/2750982721 >> 172.21.15.77:0/2750982721 conn(0x7fe14c0fcdd0 msgr2=0x7fe14c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe151b5d700 1 -- 172.21.15.77:0/2750982721 shutdown_connections 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe151b5d700 1 -- 172.21.15.77:0/2750982721 wait complete. 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe151b5d700 1 Processor -- start 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe151b5d700 1 -- start start 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe151b5d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c111870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:40.218 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe151b5d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe14c111db0 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe14b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe14b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c111870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40380/0 (socket says 172.21.15.77:40380) 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.615+0000 7fe14b7fe700 1 -- 172.21.15.77:0/3646687046 learned_addr learned my addr 172.21.15.77:0/3646687046 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.616+0000 7fe14b7fe700 1 -- 172.21.15.77:0/3646687046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe13c0094f0 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.616+0000 7fe14b7fe700 1 --2- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c111870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe13c006b00 tx=0x7fe13c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.616+0000 7fe133fff700 1 -- 172.21.15.77:0/3646687046 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe13c0109a0 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.616+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe14c111fb0 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.616+0000 7fe133fff700 1 -- 172.21.15.77:0/3646687046 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe13c01a430 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.616+0000 7fe133fff700 1 -- 172.21.15.77:0/3646687046 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe13c0245b0 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.616+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14c112c50 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.617+0000 7fe133fff700 1 -- 172.21.15.77:0/3646687046 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe13c0247e0 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.617+0000 7fe133fff700 1 --2- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe1340367e0 0x7fe134038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.617+0000 7fe133fff700 1 -- 172.21.15.77:0/3646687046 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe13c01aaa0 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.617+0000 7fe14affd700 1 --2- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe1340367e0 0x7fe134038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.617+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14c112440 con 0x7fe14c101670 2024-02-19T18:50:40.219 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.617+0000 7fe14affd700 1 --2- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe1340367e0 0x7fe134038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe140009990 tx=0x7fe140006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.622+0000 7fe133fff700 1 -- 172.21.15.77:0/3646687046 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe13c02e440 con 0x7fe14c101670 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:38.855+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fe14c1034b0 con 0x7fe14c101670 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.631+0000 7fe133fff700 1 -- 172.21.15.77:0/3646687046 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fe13c00fdc0 con 0x7fe14c101670 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.633+0000 7fe133fff700 1 -- 172.21.15.77:0/3646687046 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fe13c00f930 con 0x7fe14c101670 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe1340367e0 msgr2=0x7fe134038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 --2- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe1340367e0 0x7fe134038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe140009990 tx=0x7fe140006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 msgr2=0x7fe14c111870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 --2- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c111870 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe13c006b00 tx=0x7fe13c009250 comp rx=0 tx=0).stop 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 shutdown_connections 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 --2- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7fe1340367e0 0x7fe134038c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 --2- 172.21.15.77:0/3646687046 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe14c101670 0x7fe14c111870 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 >> 172.21.15.77:0/3646687046 conn(0x7fe14c0fcdd0 msgr2=0x7fe14c0fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 shutdown_connections 2024-02-19T18:50:40.220 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:39.636+0000 7fe151b5d700 1 -- 172.21.15.77:0/3646687046 wait complete. 2024-02-19T18:50:40.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:40 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3646687046' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-19T18:50:40.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:40 smithi077 ceph-mon[29247]: mgrmap e9: smithi077.yiiqpd(active, since 24s) 2024-02-19T18:50:41.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:50:41.490 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "active_name": "smithi077.yiiqpd", 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.610+0000 7f7591134700 1 Processor -- start 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.610+0000 7f7591134700 1 -- start start 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.610+0000 7f7591134700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.610+0000 7f7591134700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c105960 con 0x7f758c076b80 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.610+0000 7f758ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.610+0000 7f758ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40404/0 (socket says 172.21.15.77:40404) 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.610+0000 7f758ad9d700 1 -- 172.21.15.77:0/899026295 learned_addr learned my addr 172.21.15.77:0/899026295 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.610+0000 7f758ad9d700 1 -- 172.21.15.77:0/899026295 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c105aa0 con 0x7f758c076b80 2024-02-19T18:50:41.491 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.611+0000 7f758ad9d700 1 --2- 172.21.15.77:0/899026295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c105420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f757c0098e0 tx=0x7f757c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a01c3ba6bab2230d server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.611+0000 7f7589d9b700 1 -- 172.21.15.77:0/899026295 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757c00b860 con 0x7f758c076b80 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.611+0000 7f7589d9b700 1 -- 172.21.15.77:0/899026295 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f757c00be90 con 0x7f758c076b80 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.611+0000 7f7589d9b700 1 -- 172.21.15.77:0/899026295 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757c0103c0 con 0x7f758c076b80 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 -- 172.21.15.77:0/899026295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 msgr2=0x7f758c105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 --2- 172.21.15.77:0/899026295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c105420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f757c0098e0 tx=0x7f757c009250 comp rx=0 tx=0).stop 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 -- 172.21.15.77:0/899026295 shutdown_connections 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 --2- 172.21.15.77:0/899026295 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c105420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 -- 172.21.15.77:0/899026295 >> 172.21.15.77:0/899026295 conn(0x7f758c0fcdd0 msgr2=0x7f758c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 -- 172.21.15.77:0/899026295 shutdown_connections 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 -- 172.21.15.77:0/899026295 wait complete. 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 Processor -- start 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.612+0000 7f7591134700 1 -- start start 2024-02-19T18:50:41.492 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f7591134700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c1115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f7591134700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c111af0 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f758ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c1115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f758ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c1115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40416/0 (socket says 172.21.15.77:40416) 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f758ad9d700 1 -- 172.21.15.77:0/2208613968 learned_addr learned my addr 172.21.15.77:0/2208613968 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f758ad9d700 1 -- 172.21.15.77:0/2208613968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c0094f0 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f758ad9d700 1 --2- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c1115b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f757c0090b0 tx=0x7f757c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f75737fe700 1 -- 172.21.15.77:0/2208613968 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757c0118f0 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758c111cf0 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f75737fe700 1 -- 172.21.15.77:0/2208613968 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f757c010d10 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f75737fe700 1 -- 172.21.15.77:0/2208613968 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f757c01aea0 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.613+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758c112990 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.614+0000 7f75737fe700 1 -- 172.21.15.77:0/2208613968 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f757c019510 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.614+0000 7f75737fe700 1 --2- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f75740367e0 0x7f7574038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.615+0000 7f75737fe700 1 -- 172.21.15.77:0/2208613968 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f757c04a640 con 0x7f758c076b80 2024-02-19T18:50:41.493 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.615+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758c05f5c0 con 0x7f758c076b80 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.615+0000 7f758a59c700 1 -- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f75740367e0 msgr2=0x7f7574038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1028404574 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.615+0000 7f758a59c700 1 --2- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f75740367e0 0x7f7574038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.619+0000 7f75737fe700 1 -- 172.21.15.77:0/2208613968 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f757c02d3f0 con 0x7f758c076b80 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.816+0000 7f758a59c700 1 -- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f75740367e0 msgr2=0x7f7574038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1028404574 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.816+0000 7f758a59c700 1 --2- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f75740367e0 0x7f7574038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.863+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f758c112c70 con 0x7f758c076b80 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.864+0000 7f75737fe700 1 -- 172.21.15.77:0/2208613968 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f757c022330 con 0x7f758c076b80 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f75740367e0 msgr2=0x7f7574038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 --2- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f75740367e0 0x7f7574038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 msgr2=0x7f758c1115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 --2- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c1115b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f757c0090b0 tx=0x7f757c009250 comp rx=0 tx=0).stop 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 shutdown_connections 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 --2- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7f75740367e0 0x7f7574038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 --2- 172.21.15.77:0/2208613968 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f758c076b80 0x7f758c1115b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 >> 172.21.15.77:0/2208613968 conn(0x7f758c0fcdd0 msgr2=0x7f758c0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:41.494 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 shutdown_connections 2024-02-19T18:50:41.495 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:40.867+0000 7f7591134700 1 -- 172.21.15.77:0/2208613968 wait complete. 2024-02-19T18:50:41.495 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for the mgr to restart... 2024-02-19T18:50:41.495 INFO:teuthology.orchestra.run.smithi077.stdout:Waiting for mgr epoch 9... 2024-02-19T18:50:41.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:41 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/2208613968' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-19T18:50:46.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: Active manager daemon smithi077.yiiqpd restarted 2024-02-19T18:50:46.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: Activating manager daemon smithi077.yiiqpd 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: osdmap e3: 0 total, 0 up, 0 in 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: mgrmap e10: smithi077.yiiqpd(active, starting, since 0.00367717s) 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi077.yiiqpd", "id": "smithi077.yiiqpd"}]: dispatch 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: Manager daemon smithi077.yiiqpd is now available 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:50:46.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:46 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:50:47.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:47 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:50:47.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:47 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/trash_purge_schedule"}]: dispatch 2024-02-19T18:50:47.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:47 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:47.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:47 smithi077 ceph-mon[29247]: mgrmap e11: smithi077.yiiqpd(active, since 1.00525s) 2024-02-19T18:50:48.118 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout { 2024-02-19T18:50:48.118 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout } 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.901+0000 7ff081785700 1 Processor -- start 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.901+0000 7ff081785700 1 -- start start 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.901+0000 7ff081785700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c105380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.901+0000 7ff081785700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c1058c0 con 0x7ff07c071480 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.901+0000 7ff07affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c105380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.901+0000 7ff07affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c105380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40426/0 (socket says 172.21.15.77:40426) 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.901+0000 7ff07affd700 1 -- 172.21.15.77:0/1479995044 learned_addr learned my addr 172.21.15.77:0/1479995044 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.902+0000 7ff07affd700 1 -- 172.21.15.77:0/1479995044 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07c105a00 con 0x7ff07c071480 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.902+0000 7ff07affd700 1 --2- 172.21.15.77:0/1479995044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c105380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff074009e50 tx=0x7ff074009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82773895386ceb70 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.902+0000 7ff079ffb700 1 -- 172.21.15.77:0/1479995044 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0740092e0 con 0x7ff07c071480 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.902+0000 7ff079ffb700 1 -- 172.21.15.77:0/1479995044 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff074012430 con 0x7ff07c071480 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.902+0000 7ff079ffb700 1 -- 172.21.15.77:0/1479995044 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0740075a0 con 0x7ff07c071480 2024-02-19T18:50:48.119 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 -- 172.21.15.77:0/1479995044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 msgr2=0x7ff07c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 --2- 172.21.15.77:0/1479995044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c105380 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff074009e50 tx=0x7ff074009840 comp rx=0 tx=0).stop 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 -- 172.21.15.77:0/1479995044 shutdown_connections 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 --2- 172.21.15.77:0/1479995044 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c105380 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 -- 172.21.15.77:0/1479995044 >> 172.21.15.77:0/1479995044 conn(0x7ff07c0fcdd0 msgr2=0x7ff07c0ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 -- 172.21.15.77:0/1479995044 shutdown_connections 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 -- 172.21.15.77:0/1479995044 wait complete. 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 Processor -- start 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.903+0000 7ff081785700 1 -- start start 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff081785700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c1115f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff081785700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c111b30 con 0x7ff07c071480 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff07affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c1115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff07affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c1115f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:40438/0 (socket says 172.21.15.77:40438) 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff07affd700 1 -- 172.21.15.77:0/1100841261 learned_addr learned my addr 172.21.15.77:0/1100841261 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:48.120 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff07affd700 1 -- 172.21.15.77:0/1100841261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff074009ae0 con 0x7ff07c071480 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff07affd700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c1115f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff074012de0 tx=0x7ff074009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0740092e0 con 0x7ff07c071480 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff07c111d30 con 0x7ff07c071480 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff074007d30 con 0x7ff07c071480 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff074025a00 con 0x7ff07c071480 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.904+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c1129d0 con 0x7ff07c071480 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.905+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7ff074019050 con 0x7ff07c071480 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.905+0000 7ff0637fe700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 0x7ff064038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.905+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 --> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0640393a0 con 0x7ff0640367e0 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.906+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff07404a020 con 0x7ff07c071480 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.906+0000 7ff07a7fc700 1 -- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 msgr2=0x7ff064038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1028404574 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:41.906+0000 7ff07a7fc700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 0x7ff064038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:42.107+0000 7ff07a7fc700 1 -- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 msgr2=0x7ff064038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1028404574 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:42.107+0000 7ff07a7fc700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 0x7ff064038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:50:48.121 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:42.508+0000 7ff07a7fc700 1 -- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 msgr2=0x7ff064038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1028404574 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:42.508+0000 7ff07a7fc700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 0x7ff064038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:43.309+0000 7ff07a7fc700 1 -- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 msgr2=0x7ff064038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1028404574 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:43.309+0000 7ff07a7fc700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 0x7ff064038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:44.910+0000 7ff07a7fc700 1 -- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 msgr2=0x7ff064038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/1028404574 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:44.910+0000 7ff07a7fc700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 0x7ff064038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:46.167+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7ff07401c410 con 0x7ff07c071480 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:46.167+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 msgr2=0x7ff064038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:46.167+0000 7ff0637fe700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 0x7ff064038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.168+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7ff07404b950 con 0x7ff07c071480 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.168+0000 7ff0637fe700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff064039fd0 0x7ff06403c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.168+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 --> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0640393a0 con 0x7ff064039fd0 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.169+0000 7ff07a7fc700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff064039fd0 0x7ff06403c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.169+0000 7ff07a7fc700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff064039fd0 0x7ff06403c3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff0700039d0 tx=0x7ff0700092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.169+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mgr.14162 v2:172.21.15.77:6800/3287530536 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7ff0640393a0 con 0x7ff064039fd0 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.175+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 --> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff07c1121c0 con 0x7ff064039fd0 2024-02-19T18:50:48.122 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff0637fe700 1 -- 172.21.15.77:0/1100841261 <== mgr.14162 v2:172.21.15.77:6800/3287530536 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7ff07c1121c0 con 0x7ff064039fd0 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff064039fd0 msgr2=0x7ff06403c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff064039fd0 0x7ff06403c3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff0700039d0 tx=0x7ff0700092f0 comp rx=0 tx=0).stop 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 msgr2=0x7ff07c1115f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c1115f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff074012de0 tx=0x7ff074009840 comp rx=0 tx=0).stop 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 shutdown_connections 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff064039fd0 0x7ff06403c3b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff07c071480 0x7ff07c1115f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 --2- 172.21.15.77:0/1100841261 >> [v2:172.21.15.77:6800/1028404574,v1:172.21.15.77:6801/1028404574] conn(0x7ff0640367e0 0x7ff064038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 >> 172.21.15.77:0/1100841261 conn(0x7ff07c0fcdd0 msgr2=0x7ff07c0fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 shutdown_connections 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:47.176+0000 7ff081785700 1 -- 172.21.15.77:0/1100841261 wait complete. 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:mgr epoch 9 is available 2024-02-19T18:50:48.123 INFO:teuthology.orchestra.run.smithi077.stdout:Generating a dashboard self-signed certificate... 2024-02-19T18:50:48.771 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:48 smithi077 ceph-mon[29247]: [19/Feb/2024:18:50:46] ENGINE Bus STARTING 2024-02-19T18:50:48.772 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:48 smithi077 ceph-mon[29247]: [19/Feb/2024:18:50:46] ENGINE Serving on https://172.21.15.77:7150 2024-02-19T18:50:48.772 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:48 smithi077 ceph-mon[29247]: [19/Feb/2024:18:50:46] ENGINE Bus STARTED 2024-02-19T18:50:48.772 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:48 smithi077 ceph-mon[29247]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-19T18:50:48.772 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:48 smithi077 ceph-mon[29247]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-19T18:50:48.772 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:48 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:49.650 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-19T18:50:49.650 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.545+0000 7efd751ab700 1 Processor -- start 2024-02-19T18:50:49.650 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.545+0000 7efd751ab700 1 -- start start 2024-02-19T18:50:49.650 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.545+0000 7efd751ab700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70103070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:49.650 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.545+0000 7efd751ab700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd701035b0 con 0x7efd70102c60 2024-02-19T18:50:49.650 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.545+0000 7efd6ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70103070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:49.650 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.545+0000 7efd6ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70103070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51566/0 (socket says 172.21.15.77:51566) 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.545+0000 7efd6ed9d700 1 -- 172.21.15.77:0/1124384755 learned_addr learned my addr 172.21.15.77:0/1124384755 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.546+0000 7efd6ed9d700 1 -- 172.21.15.77:0/1124384755 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd701036f0 con 0x7efd70102c60 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.546+0000 7efd6ed9d700 1 --2- 172.21.15.77:0/1124384755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70103070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd600098e0 tx=0x7efd60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4ef1f4f4a4ab49 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.546+0000 7efd6dd9b700 1 -- 172.21.15.77:0/1124384755 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd6000b860 con 0x7efd70102c60 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.546+0000 7efd6dd9b700 1 -- 172.21.15.77:0/1124384755 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd6000be90 con 0x7efd70102c60 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.546+0000 7efd6dd9b700 1 -- 172.21.15.77:0/1124384755 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd600103c0 con 0x7efd70102c60 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 -- 172.21.15.77:0/1124384755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 msgr2=0x7efd70103070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 --2- 172.21.15.77:0/1124384755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70103070 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd600098e0 tx=0x7efd60009250 comp rx=0 tx=0).stop 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 -- 172.21.15.77:0/1124384755 shutdown_connections 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 --2- 172.21.15.77:0/1124384755 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70103070 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 -- 172.21.15.77:0/1124384755 >> 172.21.15.77:0/1124384755 conn(0x7efd700fc580 msgr2=0x7efd700fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:49.651 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 -- 172.21.15.77:0/1124384755 shutdown_connections 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 -- 172.21.15.77:0/1124384755 wait complete. 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 Processor -- start 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.547+0000 7efd751ab700 1 -- start start 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd751ab700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70114550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd751ab700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd70101540 con 0x7efd70102c60 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd6ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70114550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd6ed9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70114550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51578/0 (socket says 172.21.15.77:51578) 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd6ed9d700 1 -- 172.21.15.77:0/1586729862 learned_addr learned my addr 172.21.15.77:0/1586729862 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd6ed9d700 1 -- 172.21.15.77:0/1586729862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd600094f0 con 0x7efd70102c60 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd6ed9d700 1 --2- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70114550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd60006b00 tx=0x7efd60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd577fe700 1 -- 172.21.15.77:0/1586729862 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd600109a0 con 0x7efd70102c60 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd70101740 con 0x7efd70102c60 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd577fe700 1 -- 172.21.15.77:0/1586729862 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efd6001a430 con 0x7efd70102c60 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd577fe700 1 -- 172.21.15.77:0/1586729862 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd600245b0 con 0x7efd70102c60 2024-02-19T18:50:49.652 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.548+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd70114c50 con 0x7efd70102c60 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.549+0000 7efd577fe700 1 -- 172.21.15.77:0/1586729862 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7efd60024830 con 0x7efd70102c60 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.549+0000 7efd577fe700 1 --2- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7efd580366b0 0x7efd58038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.549+0000 7efd577fe700 1 -- 172.21.15.77:0/1586729862 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efd6004b6c0 con 0x7efd70102c60 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.550+0000 7efd6e59c700 1 --2- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7efd580366b0 0x7efd58038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.550+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7010a570 con 0x7efd70102c60 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.550+0000 7efd6e59c700 1 --2- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7efd580366b0 0x7efd58038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efd64009990 tx=0x7efd64006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.558+0000 7efd577fe700 1 -- 172.21.15.77:0/1586729862 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7efd60018db0 con 0x7efd70102c60 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.756+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 --> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7efd701025d0 con 0x7efd580366b0 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.782+0000 7efd577fe700 1 -- 172.21.15.77:0/1586729862 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7efd6000fdc0 con 0x7efd70102c60 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.935+0000 7efd577fe700 1 -- 172.21.15.77:0/1586729862 <== mgr.14162 v2:172.21.15.77:6800/3287530536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7efd701025d0 con 0x7efd580366b0 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.937+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7efd580366b0 msgr2=0x7efd58038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.937+0000 7efd751ab700 1 --2- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7efd580366b0 0x7efd58038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7efd64009990 tx=0x7efd64006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.937+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 msgr2=0x7efd70114550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.937+0000 7efd751ab700 1 --2- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70114550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd60006b00 tx=0x7efd60009250 comp rx=0 tx=0).stop 2024-02-19T18:50:49.653 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.937+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 shutdown_connections 2024-02-19T18:50:49.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.937+0000 7efd751ab700 1 --2- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7efd580366b0 0x7efd58038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:49.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.937+0000 7efd751ab700 1 --2- 172.21.15.77:0/1586729862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7efd70102c60 0x7efd70114550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:49.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.937+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 >> 172.21.15.77:0/1586729862 conn(0x7efd700fc580 msgr2=0x7efd700fd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:49.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.938+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 shutdown_connections 2024-02-19T18:50:49.654 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:48.938+0000 7efd751ab700 1 -- 172.21.15.77:0/1586729862 wait complete. 2024-02-19T18:50:49.654 INFO:teuthology.orchestra.run.smithi077.stdout:Creating initial admin user... 2024-02-19T18:50:49.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:49 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:49.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:49 smithi077 ceph-mon[29247]: mgrmap e12: smithi077.yiiqpd(active, since 2s) 2024-02-19T18:50:49.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:49 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:49.760 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:49 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:51.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:50 smithi077 ceph-mon[29247]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:51.810 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$rWI4iXTXXWmYpsshvb.3YO6wkZCgwMw3Y4/wh/fTcE2OALfk.EjT2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1708368651, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-19T18:50:51.810 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d8ac0700 1 Processor -- start 2024-02-19T18:50:51.810 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d8ac0700 1 -- start start 2024-02-19T18:50:51.810 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d8ac0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d41031a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:51.810 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d8ac0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d41036e0 con 0x7fc8d4102d90 2024-02-19T18:50:51.810 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d41031a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d41031a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51594/0 (socket says 172.21.15.77:51594) 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d259c700 1 -- 172.21.15.77:0/3267319228 learned_addr learned my addr 172.21.15.77:0/3267319228 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d259c700 1 -- 172.21.15.77:0/3267319228 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d4103820 con 0x7fc8d4102d90 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.821+0000 7fc8d259c700 1 --2- 172.21.15.77:0/3267319228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d41031a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80098e0 tx=0x7fc8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77fed9f21966a16 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d159a700 1 -- 172.21.15.77:0/3267319228 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8c800b860 con 0x7fc8d4102d90 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d159a700 1 -- 172.21.15.77:0/3267319228 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8c800be90 con 0x7fc8d4102d90 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d159a700 1 -- 172.21.15.77:0/3267319228 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8c80103c0 con 0x7fc8d4102d90 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/3267319228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 msgr2=0x7fc8d41031a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d8ac0700 1 --2- 172.21.15.77:0/3267319228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d41031a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80098e0 tx=0x7fc8c8009250 comp rx=0 tx=0).stop 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/3267319228 shutdown_connections 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d8ac0700 1 --2- 172.21.15.77:0/3267319228 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d41031a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/3267319228 >> 172.21.15.77:0/3267319228 conn(0x7fc8d40fc580 msgr2=0x7fc8d40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/3267319228 shutdown_connections 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.822+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/3267319228 wait complete. 2024-02-19T18:50:51.811 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d8ac0700 1 Processor -- start 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d8ac0700 1 -- start start 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d8ac0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d4113a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d8ac0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d4113f80 con 0x7fc8d4102d90 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d4113a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d4113a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51608/0 (socket says 172.21.15.77:51608) 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d259c700 1 -- 172.21.15.77:0/1498363724 learned_addr learned my addr 172.21.15.77:0/1498363724 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d259c700 1 -- 172.21.15.77:0/1498363724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8c80094f0 con 0x7fc8d4102d90 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.823+0000 7fc8d259c700 1 --2- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d4113a40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8006b00 tx=0x7fc8c8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.824+0000 7fc8c2ffd700 1 -- 172.21.15.77:0/1498363724 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8c80109a0 con 0x7fc8d4102d90 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.824+0000 7fc8c2ffd700 1 -- 172.21.15.77:0/1498363724 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8c801a430 con 0x7fc8d4102d90 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.824+0000 7fc8c2ffd700 1 -- 172.21.15.77:0/1498363724 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8c80245b0 con 0x7fc8d4102d90 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.824+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d4114180 con 0x7fc8d4102d90 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.824+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d4114e20 con 0x7fc8d4102d90 2024-02-19T18:50:51.812 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.824+0000 7fc8c2ffd700 1 -- 172.21.15.77:0/1498363724 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fc8c801aaa0 con 0x7fc8d4102d90 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.825+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d405f5c0 con 0x7fc8d4102d90 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.825+0000 7fc8c2ffd700 1 --2- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc8bc036830 0x7fc8bc038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.825+0000 7fc8d1d9b700 1 --2- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc8bc036830 0x7fc8bc038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.825+0000 7fc8c2ffd700 1 -- 172.21.15.77:0/1498363724 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc8c804a7b0 con 0x7fc8d4102d90 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.825+0000 7fc8d1d9b700 1 --2- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc8bc036830 0x7fc8bc038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009990 tx=0x7fc8c4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:50.829+0000 7fc8c2ffd700 1 -- 172.21.15.77:0/1498363724 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc8c802c340 con 0x7fc8d4102d90 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.025+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 --> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc8d405f3e0 con 0x7fc8bc036830 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.247+0000 7fc8c2ffd700 1 -- 172.21.15.77:0/1498363724 <== mgr.14162 v2:172.21.15.77:6800/3287530536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fc8d405f3e0 con 0x7fc8bc036830 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc8bc036830 msgr2=0x7fc8bc038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 --2- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc8bc036830 0x7fc8bc038ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc8c4009990 tx=0x7fc8c4006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 msgr2=0x7fc8d4113a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 --2- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d4113a40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8006b00 tx=0x7fc8c8009250 comp rx=0 tx=0).stop 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 shutdown_connections 2024-02-19T18:50:51.813 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 --2- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc8bc036830 0x7fc8bc038ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:51.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 --2- 172.21.15.77:0/1498363724 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc8d4102d90 0x7fc8d4113a40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:51.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 >> 172.21.15.77:0/1498363724 conn(0x7fc8d40fc580 msgr2=0x7fc8d40fd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:51.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 shutdown_connections 2024-02-19T18:50:51.814 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:51.249+0000 7fc8d8ac0700 1 -- 172.21.15.77:0/1498363724 wait complete. 2024-02-19T18:50:51.814 INFO:teuthology.orchestra.run.smithi077.stdout:Fetching dashboard port number... 2024-02-19T18:50:52.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:52 smithi077 ceph-mon[29247]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:50:52.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:52 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stdout 8443 2024-02-19T18:50:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.194+0000 7f3b16ee4700 1 Processor -- start 2024-02-19T18:50:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.194+0000 7f3b16ee4700 1 -- start start 2024-02-19T18:50:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.194+0000 7f3b16ee4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10101a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:52.967 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.194+0000 7f3b16ee4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b10101fc0 con 0x7f3b10101670 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.195+0000 7f3b14c80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10101a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.195+0000 7f3b14c80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10101a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51624/0 (socket says 172.21.15.77:51624) 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.195+0000 7f3b14c80700 1 -- 172.21.15.77:0/1924555775 learned_addr learned my addr 172.21.15.77:0/1924555775 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.195+0000 7f3b14c80700 1 -- 172.21.15.77:0/1924555775 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b10102100 con 0x7f3b10101670 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.195+0000 7f3b14c80700 1 --2- 172.21.15.77:0/1924555775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10101a80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009e50 tx=0x7f3b00009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15ec181fa6976fbe server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.195+0000 7f3b0f7fe700 1 -- 172.21.15.77:0/1924555775 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b000092e0 con 0x7f3b10101670 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.195+0000 7f3b0f7fe700 1 -- 172.21.15.77:0/1924555775 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b00012430 con 0x7f3b10101670 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.195+0000 7f3b0f7fe700 1 -- 172.21.15.77:0/1924555775 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b000075a0 con 0x7f3b10101670 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.196+0000 7f3b16ee4700 1 -- 172.21.15.77:0/1924555775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 msgr2=0x7f3b10101a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.196+0000 7f3b16ee4700 1 --2- 172.21.15.77:0/1924555775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10101a80 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3b00009e50 tx=0x7f3b00009840 comp rx=0 tx=0).stop 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.196+0000 7f3b16ee4700 1 -- 172.21.15.77:0/1924555775 shutdown_connections 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.196+0000 7f3b16ee4700 1 --2- 172.21.15.77:0/1924555775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10101a80 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.196+0000 7f3b16ee4700 1 -- 172.21.15.77:0/1924555775 >> 172.21.15.77:0/1924555775 conn(0x7f3b100fcdd0 msgr2=0x7f3b100ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:52.968 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.196+0000 7f3b16ee4700 1 -- 172.21.15.77:0/1924555775 shutdown_connections 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.196+0000 7f3b16ee4700 1 -- 172.21.15.77:0/1924555775 wait complete. 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b16ee4700 1 Processor -- start 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b16ee4700 1 -- start start 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b16ee4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10114270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b16ee4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b10076c50 con 0x7f3b10101670 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b14c80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10114270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b14c80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10114270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51638/0 (socket says 172.21.15.77:51638) 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b14c80700 1 -- 172.21.15.77:0/4166067239 learned_addr learned my addr 172.21.15.77:0/4166067239 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b14c80700 1 -- 172.21.15.77:0/4166067239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b00009ae0 con 0x7f3b10101670 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b14c80700 1 --2- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10114270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b00012800 tx=0x7f3b00009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.197+0000 7f3b0d7fa700 1 -- 172.21.15.77:0/4166067239 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b000092e0 con 0x7f3b10101670 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.198+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b10076e50 con 0x7f3b10101670 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.198+0000 7f3b0d7fa700 1 -- 172.21.15.77:0/4166067239 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b00007d30 con 0x7f3b10101670 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.198+0000 7f3b0d7fa700 1 -- 172.21.15.77:0/4166067239 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b000259a0 con 0x7f3b10101670 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.198+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b101149f0 con 0x7f3b10101670 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.198+0000 7f3b0d7fa700 1 -- 172.21.15.77:0/4166067239 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f3b00019050 con 0x7f3b10101670 2024-02-19T18:50:52.969 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.198+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1004bed0 con 0x7f3b10101670 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.199+0000 7f3b0d7fa700 1 --2- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3afc0367e0 0x7f3afc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.199+0000 7f3b0d7fa700 1 -- 172.21.15.77:0/4166067239 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3b0004a3a0 con 0x7f3b10101670 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.199+0000 7f3b0ffff700 1 --2- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3afc0367e0 0x7f3afc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.199+0000 7f3b0ffff700 1 --2- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3afc0367e0 0x7f3afc038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009990 tx=0x7f3af8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.203+0000 7f3b0d7fa700 1 -- 172.21.15.77:0/4166067239 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3b00023320 con 0x7f3b10101670 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.249+0000 7f3b0d7fa700 1 -- 172.21.15.77:0/4166067239 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f3b00025c10 con 0x7f3b10101670 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.390+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f3b101034b0 con 0x7f3b10101670 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.391+0000 7f3b0d7fa700 1 -- 172.21.15.77:0/4166067239 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f3b00023320 con 0x7f3b10101670 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3afc0367e0 msgr2=0x7f3afc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 --2- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3afc0367e0 0x7f3afc038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3af8009990 tx=0x7f3af8006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 msgr2=0x7f3b10114270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 --2- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10114270 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3b00012800 tx=0x7f3b00009840 comp rx=0 tx=0).stop 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 shutdown_connections 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 --2- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3afc0367e0 0x7f3afc038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:52.970 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 --2- 172.21.15.77:0/4166067239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3b10101670 0x7f3b10114270 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:52.971 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 >> 172.21.15.77:0/4166067239 conn(0x7f3b100fcdd0 msgr2=0x7f3b100fe4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:52.971 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 shutdown_connections 2024-02-19T18:50:52.971 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:52.393+0000 7f3b16ee4700 1 -- 172.21.15.77:0/4166067239 wait complete. 2024-02-19T18:50:52.973 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-19T18:50:52.973 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout disabled 2024-02-19T18:50:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-19T18:50:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:systemctl: stdout inactive 2024-02-19T18:50:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:firewalld.service is not enabled 2024-02-19T18:50:52.980 INFO:teuthology.orchestra.run.smithi077.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-19T18:50:52.982 INFO:teuthology.orchestra.run.smithi077.stdout:Ceph Dashboard is now available at: 2024-02-19T18:50:52.982 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:52.982 INFO:teuthology.orchestra.run.smithi077.stdout: URL: https://smithi077.front.sepia.ceph.com:8443/ 2024-02-19T18:50:52.982 INFO:teuthology.orchestra.run.smithi077.stdout: User: admin 2024-02-19T18:50:52.982 INFO:teuthology.orchestra.run.smithi077.stdout: Password: 72ux40l8yh 2024-02-19T18:50:52.983 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:52.983 INFO:teuthology.orchestra.run.smithi077.stdout:Saving cluster configuration to /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config directory 2024-02-19T18:50:52.983 INFO:teuthology.orchestra.run.smithi077.stdout:Enabling autotune for osd_memory_target 2024-02-19T18:50:53.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:53 smithi077 ceph-mon[29247]: mgrmap e13: smithi077.yiiqpd(active, since 6s) 2024-02-19T18:50:53.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:53 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4166067239' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-19T18:50:54.106 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.369+0000 7fcea6058700 1 Processor -- start 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.369+0000 7fcea6058700 1 -- start start 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.369+0000 7fcea6058700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea0102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.369+0000 7fcea6058700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea01030c0 con 0x7fcea0102770 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.369+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea0102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.370+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea0102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51644/0 (socket says 172.21.15.77:51644) 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.370+0000 7fce9f7fe700 1 -- 172.21.15.77:0/1977746896 learned_addr learned my addr 172.21.15.77:0/1977746896 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.370+0000 7fce9f7fe700 1 -- 172.21.15.77:0/1977746896 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea0103200 con 0x7fcea0102770 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.370+0000 7fce9f7fe700 1 --2- 172.21.15.77:0/1977746896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea0102b80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fce900098e0 tx=0x7fce90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36519b51763d286 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.370+0000 7fce9e7fc700 1 -- 172.21.15.77:0/1977746896 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce9000b860 con 0x7fcea0102770 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.370+0000 7fce9e7fc700 1 -- 172.21.15.77:0/1977746896 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce9000be90 con 0x7fcea0102770 2024-02-19T18:50:54.107 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.370+0000 7fce9e7fc700 1 -- 172.21.15.77:0/1977746896 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce900103c0 con 0x7fcea0102770 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 -- 172.21.15.77:0/1977746896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 msgr2=0x7fcea0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 --2- 172.21.15.77:0/1977746896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea0102b80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fce900098e0 tx=0x7fce90009250 comp rx=0 tx=0).stop 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 -- 172.21.15.77:0/1977746896 shutdown_connections 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 --2- 172.21.15.77:0/1977746896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea0102b80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 -- 172.21.15.77:0/1977746896 >> 172.21.15.77:0/1977746896 conn(0x7fcea00fded0 msgr2=0x7fcea0100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 -- 172.21.15.77:0/1977746896 shutdown_connections 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 -- 172.21.15.77:0/1977746896 wait complete. 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 Processor -- start 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.371+0000 7fcea6058700 1 -- start start 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fcea6058700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea01115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fcea6058700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0111b00 con 0x7fcea0102770 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea01115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:54.108 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea01115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51652/0 (socket says 172.21.15.77:51652) 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fce9f7fe700 1 -- 172.21.15.77:0/83053812 learned_addr learned my addr 172.21.15.77:0/83053812 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fce9f7fe700 1 -- 172.21.15.77:0/83053812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce900094f0 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fce9f7fe700 1 --2- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea01115c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fce900090b0 tx=0x7fce90009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fce87fff700 1 -- 172.21.15.77:0/83053812 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce90010840 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea0111d00 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fce87fff700 1 -- 172.21.15.77:0/83053812 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce90010e70 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fce87fff700 1 -- 172.21.15.77:0/83053812 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce90024e40 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.372+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea01129a0 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.373+0000 7fce87fff700 1 -- 172.21.15.77:0/83053812 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fce9001a530 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.373+0000 7fce87fff700 1 --2- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fce88036830 0x7fce88038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.373+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea005f5c0 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.373+0000 7fce87fff700 1 -- 172.21.15.77:0/83053812 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fce9004abd0 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.373+0000 7fce9effd700 1 --2- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fce88036830 0x7fce88038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.374+0000 7fce9effd700 1 --2- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fce88036830 0x7fce88038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fce94009990 tx=0x7fce94006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.378+0000 7fce87fff700 1 -- 172.21.15.77:0/83053812 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fce90018530 con 0x7fcea0102770 2024-02-19T18:50:54.109 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.560+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fcea004bed0 con 0x7fcea0102770 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.561+0000 7fce87fff700 1 -- 172.21.15.77:0/83053812 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fce90018350 con 0x7fcea0102770 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.562+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fce88036830 msgr2=0x7fce88038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.562+0000 7fcea6058700 1 --2- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fce88036830 0x7fce88038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fce94009990 tx=0x7fce94006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.562+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 msgr2=0x7fcea01115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.562+0000 7fcea6058700 1 --2- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea01115c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fce900090b0 tx=0x7fce90009250 comp rx=0 tx=0).stop 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.563+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 shutdown_connections 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.563+0000 7fcea6058700 1 --2- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fce88036830 0x7fce88038ce0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.563+0000 7fcea6058700 1 --2- 172.21.15.77:0/83053812 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcea0102770 0x7fcea01115c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.563+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 >> 172.21.15.77:0/83053812 conn(0x7fcea00fded0 msgr2=0x7fcea00ff5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.563+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 shutdown_connections 2024-02-19T18:50:54.110 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:53.563+0000 7fcea6058700 1 -- 172.21.15.77:0/83053812 wait complete. 2024-02-19T18:50:55.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.498+0000 7f943058b700 1 Processor -- start 2024-02-19T18:50:55.381 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.498+0000 7f943058b700 1 -- start start 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.498+0000 7f943058b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f9428102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.498+0000 7f943058b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94281030c0 con 0x7f9428102770 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.499+0000 7f942e327700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f9428102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.499+0000 7f942e327700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f9428102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51658/0 (socket says 172.21.15.77:51658) 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.499+0000 7f942e327700 1 -- 172.21.15.77:0/3173700186 learned_addr learned my addr 172.21.15.77:0/3173700186 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.499+0000 7f942e327700 1 -- 172.21.15.77:0/3173700186 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9428103200 con 0x7f9428102770 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.499+0000 7f942e327700 1 --2- 172.21.15.77:0/3173700186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f9428102b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9424009e50 tx=0x7f9424009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5d330724116bab server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.499+0000 7f942d325700 1 -- 172.21.15.77:0/3173700186 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94240092e0 con 0x7f9428102770 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.499+0000 7f942d325700 1 -- 172.21.15.77:0/3173700186 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9424012430 con 0x7f9428102770 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.499+0000 7f942d325700 1 -- 172.21.15.77:0/3173700186 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94240075a0 con 0x7f9428102770 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 -- 172.21.15.77:0/3173700186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 msgr2=0x7f9428102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 --2- 172.21.15.77:0/3173700186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f9428102b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9424009e50 tx=0x7f9424009840 comp rx=0 tx=0).stop 2024-02-19T18:50:55.382 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 -- 172.21.15.77:0/3173700186 shutdown_connections 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 --2- 172.21.15.77:0/3173700186 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f9428102b80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 -- 172.21.15.77:0/3173700186 >> 172.21.15.77:0/3173700186 conn(0x7f94280fded0 msgr2=0x7f9428100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 -- 172.21.15.77:0/3173700186 shutdown_connections 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 -- 172.21.15.77:0/3173700186 wait complete. 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 Processor -- start 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 -- start start 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f94281115b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f943058b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9428111af0 con 0x7f9428102770 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f942e327700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f94281115b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f942e327700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f94281115b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51662/0 (socket says 172.21.15.77:51662) 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.500+0000 7f942e327700 1 -- 172.21.15.77:0/3324571629 learned_addr learned my addr 172.21.15.77:0/3324571629 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.501+0000 7f942e327700 1 -- 172.21.15.77:0/3324571629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9424009ae0 con 0x7f9428102770 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.501+0000 7f942e327700 1 --2- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f94281115b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9424012800 tx=0x7f9424009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:55.383 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.501+0000 7f941effd700 1 -- 172.21.15.77:0/3324571629 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94240092e0 con 0x7f9428102770 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.501+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9428111cf0 con 0x7f9428102770 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.501+0000 7f941effd700 1 -- 172.21.15.77:0/3324571629 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9424007d30 con 0x7f9428102770 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.501+0000 7f941effd700 1 -- 172.21.15.77:0/3324571629 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94240259a0 con 0x7f9428102770 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.501+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9428112990 con 0x7f9428102770 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.502+0000 7f941effd700 1 -- 172.21.15.77:0/3324571629 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9424019050 con 0x7f9428102770 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.502+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942805f5c0 con 0x7f9428102770 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.502+0000 7f941effd700 1 --2- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f9414036880 0x7f9414038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.502+0000 7f941effd700 1 -- 172.21.15.77:0/3324571629 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f942404a3f0 con 0x7f9428102770 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.502+0000 7f942db26700 1 --2- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f9414036880 0x7f9414038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:55.384 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.502+0000 7f942db26700 1 --2- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f9414036880 0x7f9414038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9418009990 tx=0x7f9418006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.506+0000 7f941effd700 1 -- 172.21.15.77:0/3324571629 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9424011570 con 0x7f9428102770 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.790+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f942804bed0 con 0x7f9428102770 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.793+0000 7f941effd700 1 -- 172.21.15.77:0/3324571629 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f9424020070 con 0x7f9428102770 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f9414036880 msgr2=0x7f9414038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 --2- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f9414036880 0x7f9414038d30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9418009990 tx=0x7f9418006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 msgr2=0x7f94281115b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 --2- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f94281115b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9424012800 tx=0x7f9424009840 comp rx=0 tx=0).stop 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 shutdown_connections 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 --2- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f9414036880 0x7f9414038d30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 --2- 172.21.15.77:0/3324571629 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9428102770 0x7f94281115b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 >> 172.21.15.77:0/3324571629 conn(0x7f94280fded0 msgr2=0x7f94280ff5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 shutdown_connections 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr 2024-02-19T18:50:54.796+0000 7f943058b700 1 -- 172.21.15.77:0/3324571629 wait complete. 2024-02-19T18:50:55.385 INFO:teuthology.orchestra.run.smithi077.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout:Or, if you are only running a single cluster on this host: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: ceph telemetry on 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout:For more information see: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:55.386 INFO:teuthology.orchestra.run.smithi077.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-02-19T18:50:55.387 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:50:55.387 INFO:teuthology.orchestra.run.smithi077.stdout:Bootstrap complete. 2024-02-19T18:50:55.414 INFO:tasks.cephadm:Fetching config... 2024-02-19T18:50:55.415 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:50:55.415 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-19T18:50:55.469 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-19T18:50:55.469 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:50:55.470 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-19T18:50:55.524 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-19T18:50:55.524 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:50:55.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/keyring of=/dev/stdout 2024-02-19T18:50:55.591 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-19T18:50:55.591 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:50:55.592 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-19T18:50:55.646 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-19T18:50:55.646 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIFJBfDNK+jtfEcX4T56qJQTp5o+vjm8mmcglfds0U13RN3MyQmfQeym98bfMfZz9wav42zhWN0mHDojHvJBXYScB9RE0brQ8qFKypWQrGonKXYs2+TqGGJQw2cE7E7z5E25EnkiIBPbPlNwL6c0FBzOzA4CoRuc5mgpc3VlZANyq5QHWIry6magvLY5Cwq/ZfK39Ebd+Mcz73dlgkff4tx6WYBmFRbzNL/VJiruSdgufsuoxQPD/y64Lj/SYABR30mBUES678SA+2uDSx7sZAtKgE30q9Zzos3AXZDE2HKaNtY17FjMk6dBAljjBJD4oDLSjEss7wNXP7bLHbofN5qV9IWbWKgSObDZsgom7vv+F509/xhYgVQQhAEOdrzjLg1MpiHhCIqYUo/aSjNHvskqZY26b7Lt0eNb/+VTSayoVOdBcUKRUppBSRNBqyAdV6J1y6NckbIBddhKStPmcYlI2Ic9f2FhcKnzIQSC2g9tNHlkM6jbkc9dvr1UWzPLs= ceph-6778a95a-cf57-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T18:50:55.726 INFO:teuthology.orchestra.run.smithi077.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIFJBfDNK+jtfEcX4T56qJQTp5o+vjm8mmcglfds0U13RN3MyQmfQeym98bfMfZz9wav42zhWN0mHDojHvJBXYScB9RE0brQ8qFKypWQrGonKXYs2+TqGGJQw2cE7E7z5E25EnkiIBPbPlNwL6c0FBzOzA4CoRuc5mgpc3VlZANyq5QHWIry6magvLY5Cwq/ZfK39Ebd+Mcz73dlgkff4tx6WYBmFRbzNL/VJiruSdgufsuoxQPD/y64Lj/SYABR30mBUES678SA+2uDSx7sZAtKgE30q9Zzos3AXZDE2HKaNtY17FjMk6dBAljjBJD4oDLSjEss7wNXP7bLHbofN5qV9IWbWKgSObDZsgom7vv+F509/xhYgVQQhAEOdrzjLg1MpiHhCIqYUo/aSjNHvskqZY26b7Lt0eNb/+VTSayoVOdBcUKRUppBSRNBqyAdV6J1y6NckbIBddhKStPmcYlI2Ic9f2FhcKnzIQSC2g9tNHlkM6jbkc9dvr1UWzPLs= ceph-6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:50:55.740 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIFJBfDNK+jtfEcX4T56qJQTp5o+vjm8mmcglfds0U13RN3MyQmfQeym98bfMfZz9wav42zhWN0mHDojHvJBXYScB9RE0brQ8qFKypWQrGonKXYs2+TqGGJQw2cE7E7z5E25EnkiIBPbPlNwL6c0FBzOzA4CoRuc5mgpc3VlZANyq5QHWIry6magvLY5Cwq/ZfK39Ebd+Mcz73dlgkff4tx6WYBmFRbzNL/VJiruSdgufsuoxQPD/y64Lj/SYABR30mBUES678SA+2uDSx7sZAtKgE30q9Zzos3AXZDE2HKaNtY17FjMk6dBAljjBJD4oDLSjEss7wNXP7bLHbofN5qV9IWbWKgSObDZsgom7vv+F509/xhYgVQQhAEOdrzjLg1MpiHhCIqYUo/aSjNHvskqZY26b7Lt0eNb/+VTSayoVOdBcUKRUppBSRNBqyAdV6J1y6NckbIBddhKStPmcYlI2Ic9f2FhcKnzIQSC2g9tNHlkM6jbkc9dvr1UWzPLs= ceph-6778a95a-cf57-11ee-95bb-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-19T18:50:55.786 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDIFJBfDNK+jtfEcX4T56qJQTp5o+vjm8mmcglfds0U13RN3MyQmfQeym98bfMfZz9wav42zhWN0mHDojHvJBXYScB9RE0brQ8qFKypWQrGonKXYs2+TqGGJQw2cE7E7z5E25EnkiIBPbPlNwL6c0FBzOzA4CoRuc5mgpc3VlZANyq5QHWIry6magvLY5Cwq/ZfK39Ebd+Mcz73dlgkff4tx6WYBmFRbzNL/VJiruSdgufsuoxQPD/y64Lj/SYABR30mBUES678SA+2uDSx7sZAtKgE30q9Zzos3AXZDE2HKaNtY17FjMk6dBAljjBJD4oDLSjEss7wNXP7bLHbofN5qV9IWbWKgSObDZsgom7vv+F509/xhYgVQQhAEOdrzjLg1MpiHhCIqYUo/aSjNHvskqZY26b7Lt0eNb/+VTSayoVOdBcUKRUppBSRNBqyAdV6J1y6NckbIBddhKStPmcYlI2Ic9f2FhcKnzIQSC2g9tNHlkM6jbkc9dvr1UWzPLs= ceph-6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:50:55.801 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-19T18:50:55.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:55 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3324571629' entity='client.admin' 2024-02-19T18:50:56.057 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.507+0000 7f9696e09700 1 -- 172.21.15.77:0/3417045928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 msgr2=0x7f96900fbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.507+0000 7f9696e09700 1 --2- 172.21.15.77:0/3417045928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 0x7f96900fbec0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9688009c60 tx=0x7f9688004d10 comp rx=0 tx=0).stop 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.507+0000 7f9696e09700 1 -- 172.21.15.77:0/3417045928 shutdown_connections 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.507+0000 7f9696e09700 1 --2- 172.21.15.77:0/3417045928 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 0x7f96900fbec0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.507+0000 7f9696e09700 1 -- 172.21.15.77:0/3417045928 >> 172.21.15.77:0/3417045928 conn(0x7f96900f7210 msgr2=0x7f96900f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.507+0000 7f9696e09700 1 -- 172.21.15.77:0/3417045928 shutdown_connections 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.507+0000 7f9696e09700 1 -- 172.21.15.77:0/3417045928 wait complete. 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.508+0000 7f9696e09700 1 Processor -- start 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.508+0000 7f9696e09700 1 -- start start 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.508+0000 7f9696e09700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 0x7f96900f92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.508+0000 7f9696e09700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96900f7750 con 0x7f96900fbab0 2024-02-19T18:50:58.526 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.508+0000 7f9694ba5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 0x7f96900f92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.508+0000 7f9694ba5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 0x7f96900f92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41982/0 (socket says 172.21.15.77:41982) 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.508+0000 7f9694ba5700 1 -- 172.21.15.77:0/102191185 learned_addr learned my addr 172.21.15.77:0/102191185 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.508+0000 7f9694ba5700 1 -- 172.21.15.77:0/102191185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9688009940 con 0x7f96900fbab0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.509+0000 7f9694ba5700 1 --2- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 0x7f96900f92d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9688004e60 tx=0x7f9688004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.509+0000 7f968d7fa700 1 -- 172.21.15.77:0/102191185 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f968801d440 con 0x7f96900fbab0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.509+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96900f7950 con 0x7f96900fbab0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.509+0000 7f968d7fa700 1 -- 172.21.15.77:0/102191185 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f968801da70 con 0x7f96900fbab0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.509+0000 7f968d7fa700 1 -- 172.21.15.77:0/102191185 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9688012aa0 con 0x7f96900fbab0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.509+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96900f7e40 con 0x7f96900fbab0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.509+0000 7f968d7fa700 1 -- 172.21.15.77:0/102191185 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9688019040 con 0x7f96900fbab0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.510+0000 7f968d7fa700 1 --2- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f967c0368c0 0x7f967c038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.510+0000 7f968d7fa700 1 -- 172.21.15.77:0/102191185 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f968804b280 con 0x7f96900fbab0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.510+0000 7f968ffff700 1 --2- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f967c0368c0 0x7f967c038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:50:58.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.510+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690106810 con 0x7f96900fbab0 2024-02-19T18:50:58.528 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.510+0000 7f968ffff700 1 --2- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f967c0368c0 0x7f967c038d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9678009990 tx=0x7f9678006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:50:58.530 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.515+0000 7f968d7fa700 1 -- 172.21.15.77:0/102191185 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9688017080 con 0x7f96900fbab0 2024-02-19T18:50:58.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.706+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f9690106230 con 0x7f96900fbab0 2024-02-19T18:50:58.725 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.709+0000 7f968d7fa700 1 -- 172.21.15.77:0/102191185 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f9688021070 con 0x7f96900fbab0 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f967c0368c0 msgr2=0x7f967c038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 --2- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f967c0368c0 0x7f967c038d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9678009990 tx=0x7f9678006e40 comp rx=0 tx=0).stop 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 msgr2=0x7f96900f92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 --2- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 0x7f96900f92d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9688004e60 tx=0x7f9688004d10 comp rx=0 tx=0).stop 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 shutdown_connections 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 --2- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f967c0368c0 0x7f967c038d70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 --2- 172.21.15.77:0/102191185 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96900fbab0 0x7f96900f92d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 >> 172.21.15.77:0/102191185 conn(0x7f96900f7210 msgr2=0x7f9690066430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:50:58.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 shutdown_connections 2024-02-19T18:50:58.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:50:58.712+0000 7f9696e09700 1 -- 172.21.15.77:0/102191185 wait complete. 2024-02-19T18:50:59.977 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/102191185' entity='client.admin' 2024-02-19T18:50:59.977 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:59.977 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:59.977 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:50:59.978 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:50:59.978 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:50:59.978 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T18:50:59.978 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:50:59.978 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:50:59 smithi077 ceph-mon[29247]: Deploying daemon crash.smithi077 on smithi077 2024-02-19T18:51:00.003 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-19T18:51:00.003 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-19T18:51:01.243 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.781+0000 7fe290bdc700 1 -- 172.21.15.77:0/4272392797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 msgr2=0x7fe28c0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.781+0000 7fe290bdc700 1 --2- 172.21.15.77:0/4272392797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 0x7fe28c0fbed0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe2840048c0 tx=0x7fe284009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.781+0000 7fe290bdc700 1 -- 172.21.15.77:0/4272392797 shutdown_connections 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.781+0000 7fe290bdc700 1 --2- 172.21.15.77:0/4272392797 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 0x7fe28c0fbed0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.781+0000 7fe290bdc700 1 -- 172.21.15.77:0/4272392797 >> 172.21.15.77:0/4272392797 conn(0x7fe28c0f71a0 msgr2=0x7fe28c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.781+0000 7fe290bdc700 1 -- 172.21.15.77:0/4272392797 shutdown_connections 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.781+0000 7fe290bdc700 1 -- 172.21.15.77:0/4272392797 wait complete. 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.782+0000 7fe290bdc700 1 Processor -- start 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.782+0000 7fe290bdc700 1 -- start start 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.782+0000 7fe290bdc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 0x7fe28c10fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:02.798 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.782+0000 7fe290bdc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28c066810 con 0x7fe28c0fbac0 2024-02-19T18:51:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.782+0000 7fe28a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 0x7fe28c10fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.782+0000 7fe28a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 0x7fe28c10fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41990/0 (socket says 172.21.15.77:41990) 2024-02-19T18:51:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.782+0000 7fe28a59c700 1 -- 172.21.15.77:0/1260238663 learned_addr learned my addr 172.21.15.77:0/1260238663 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:51:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.782+0000 7fe28a59c700 1 -- 172.21.15.77:0/1260238663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe284005040 con 0x7fe28c0fbac0 2024-02-19T18:51:02.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.783+0000 7fe28a59c700 1 --2- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 0x7fe28c10fbc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe284009680 tx=0x7fe284009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:02.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.784+0000 7fe27affd700 1 -- 172.21.15.77:0/1260238663 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28400bed0 con 0x7fe28c0fbac0 2024-02-19T18:51:02.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.784+0000 7fe27affd700 1 -- 172.21.15.77:0/1260238663 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe28400b5c0 con 0x7fe28c0fbac0 2024-02-19T18:51:02.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.784+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28c066a10 con 0x7fe28c0fbac0 2024-02-19T18:51:02.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.784+0000 7fe27affd700 1 -- 172.21.15.77:0/1260238663 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe28401c5b0 con 0x7fe28c0fbac0 2024-02-19T18:51:02.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.784+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28c110340 con 0x7fe28c0fbac0 2024-02-19T18:51:02.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.785+0000 7fe27affd700 1 -- 172.21.15.77:0/1260238663 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fe28401c7d0 con 0x7fe28c0fbac0 2024-02-19T18:51:02.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.785+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe28c105890 con 0x7fe28c0fbac0 2024-02-19T18:51:02.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.785+0000 7fe27affd700 1 --2- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe274036870 0x7fe274038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:02.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.785+0000 7fe27affd700 1 -- 172.21.15.77:0/1260238663 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe284049a80 con 0x7fe28c0fbac0 2024-02-19T18:51:02.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.785+0000 7fe289d9b700 1 --2- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe274036870 0x7fe274038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:02.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.786+0000 7fe289d9b700 1 --2- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe274036870 0x7fe274038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe280009990 tx=0x7fe280006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:02.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.790+0000 7fe27affd700 1 -- 172.21.15.77:0/1260238663 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe284011db0 con 0x7fe28c0fbac0 2024-02-19T18:51:02.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.978+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 --> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fe28c02b540 con 0x7fe274036870 2024-02-19T18:51:02.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.981+0000 7fe27affd700 1 -- 172.21.15.77:0/1260238663 <== mgr.14162 v2:172.21.15.77:6800/3287530536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe28c02b540 con 0x7fe274036870 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe274036870 msgr2=0x7fe274038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 --2- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe274036870 0x7fe274038d20 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe280009990 tx=0x7fe280006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 msgr2=0x7fe28c10fbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 --2- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 0x7fe28c10fbc0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe284009680 tx=0x7fe284009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 shutdown_connections 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 --2- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe274036870 0x7fe274038d20 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 --2- 172.21.15.77:0/1260238663 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe28c0fbac0 0x7fe28c10fbc0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 >> 172.21.15.77:0/1260238663 conn(0x7fe28c0f71a0 msgr2=0x7fe28c0f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 shutdown_connections 2024-02-19T18:51:02.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:02.983+0000 7fe290bdc700 1 -- 172.21.15.77:0/1260238663 wait complete. 2024-02-19T18:51:03.533 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2024-02-19T18:51:03.533 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:51:03.533 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2024-02-19T18:51:03.551 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:51:03.551 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:51:03.608 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2024-02-19T18:51:03.609 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch host add smithi104 2024-02-19T18:51:03.868 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:51:04.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:03 smithi077 ceph-mon[29247]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:51:04.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:03 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:04.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:03 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:04.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:03 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:04.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:03 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:04.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:03 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.199+0000 7f458fa2d700 1 -- 172.21.15.77:0/1878672764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 msgr2=0x7f45880fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.199+0000 7f458fa2d700 1 --2- 172.21.15.77:0/1878672764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 0x7f45880fcfe0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f45840048c0 tx=0x7f4584009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.199+0000 7f458fa2d700 1 -- 172.21.15.77:0/1878672764 shutdown_connections 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.199+0000 7f458fa2d700 1 --2- 172.21.15.77:0/1878672764 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 0x7f45880fcfe0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.199+0000 7f458fa2d700 1 -- 172.21.15.77:0/1878672764 >> 172.21.15.77:0/1878672764 conn(0x7f45880f8290 msgr2=0x7f45880fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458fa2d700 1 -- 172.21.15.77:0/1878672764 shutdown_connections 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458fa2d700 1 -- 172.21.15.77:0/1878672764 wait complete. 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458fa2d700 1 Processor -- start 2024-02-19T18:51:05.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458fa2d700 1 -- start start 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458fa2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 0x7f458810cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458fa2d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458810d430 con 0x7f45880fcbd0 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458d7c9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 0x7f458810cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458d7c9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 0x7f458810cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:42008/0 (socket says 172.21.15.77:42008) 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.200+0000 7f458d7c9700 1 -- 172.21.15.77:0/3404191936 learned_addr learned my addr 172.21.15.77:0/3404191936 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.201+0000 7f458d7c9700 1 -- 172.21.15.77:0/3404191936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4584005040 con 0x7f45880fcbd0 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.201+0000 7f458d7c9700 1 --2- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 0x7f458810cef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4584009680 tx=0x7f4584009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.201+0000 7f457dffb700 1 -- 172.21.15.77:0/3404191936 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f458400bed0 con 0x7f45880fcbd0 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.201+0000 7f457dffb700 1 -- 172.21.15.77:0/3404191936 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f458400b5c0 con 0x7f45880fcbd0 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.201+0000 7f457dffb700 1 -- 172.21.15.77:0/3404191936 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f458401c5b0 con 0x7f45880fcbd0 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.201+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458810d630 con 0x7f45880fcbd0 2024-02-19T18:51:05.218 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.201+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458810e2d0 con 0x7f45880fcbd0 2024-02-19T18:51:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.202+0000 7f457dffb700 1 -- 172.21.15.77:0/3404191936 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f458401c820 con 0x7f45880fcbd0 2024-02-19T18:51:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.202+0000 7f457dffb700 1 --2- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f4574036820 0x7f4574038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:05.219 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.202+0000 7f457dffb700 1 -- 172.21.15.77:0/3404191936 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4584049ae0 con 0x7f45880fcbd0 2024-02-19T18:51:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.203+0000 7f458cfc8700 1 --2- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f4574036820 0x7f4574038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.203+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458805f5c0 con 0x7f45880fcbd0 2024-02-19T18:51:05.220 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.203+0000 7f458cfc8700 1 --2- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f4574036820 0x7f4574038cd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4578009990 tx=0x7f4578006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:05.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.207+0000 7f457dffb700 1 -- 172.21.15.77:0/3404191936 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4584023300 con 0x7f45880fcbd0 2024-02-19T18:51:05.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:04 smithi077 ceph-mon[29247]: Deploying daemon node-exporter.smithi077 on smithi077 2024-02-19T18:51:05.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:05.412+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 --> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}) v1 -- 0x7f458810e5d0 con 0x7f4574036820 2024-02-19T18:51:06.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:05 smithi077 ceph-mon[29247]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:51:07.512 INFO:teuthology.orchestra.run.smithi077.stdout:Added host 'smithi104' with addr '172.21.15.104' 2024-02-19T18:51:07.512 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.496+0000 7f457dffb700 1 -- 172.21.15.77:0/3404191936 <== mgr.14162 v2:172.21.15.77:6800/3287530536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f458810e5d0 con 0x7f4574036820 2024-02-19T18:51:07.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:07 smithi077 ceph-mon[29247]: Deploying cephadm binary to smithi104 2024-02-19T18:51:07.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:07 smithi077 ceph-mon[29247]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:07.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:07 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f4574036820 msgr2=0x7f4574038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 --2- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f4574036820 0x7f4574038cd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4578009990 tx=0x7f4578006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 msgr2=0x7f458810cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 --2- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 0x7f458810cef0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4584009680 tx=0x7f4584009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 shutdown_connections 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 --2- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f4574036820 0x7f4574038cd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 --2- 172.21.15.77:0/3404191936 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f45880fcbd0 0x7f458810cef0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 >> 172.21.15.77:0/3404191936 conn(0x7f45880f8290 msgr2=0x7f45880f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 shutdown_connections 2024-02-19T18:51:07.514 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:07.498+0000 7f458fa2d700 1 -- 172.21.15.77:0/3404191936 wait complete. 2024-02-19T18:51:08.539 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch host ls --format=json 2024-02-19T18:51:08.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:08 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:08.850 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:08 smithi077 ceph-mon[29247]: Added host smithi104 2024-02-19T18:51:09.030 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:51:09.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:09 smithi077 ceph-mon[29247]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 -- 172.21.15.77:0/4115333308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 msgr2=0x7f82180fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 --2- 172.21.15.77:0/4115333308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 0x7f82180fcfe0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f82080048c0 tx=0x7f8208009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 -- 172.21.15.77:0/4115333308 shutdown_connections 2024-02-19T18:51:11.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 --2- 172.21.15.77:0/4115333308 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 0x7f82180fcfe0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 -- 172.21.15.77:0/4115333308 >> 172.21.15.77:0/4115333308 conn(0x7f82180f8290 msgr2=0x7f82180fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 -- 172.21.15.77:0/4115333308 shutdown_connections 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 -- 172.21.15.77:0/4115333308 wait complete. 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 Processor -- start 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 -- start start 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 0x7f8218071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.121+0000 7f821d53d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82180702e0 con 0x7f82180fcbd0 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f8216ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 0x7f8218071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f8216ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 0x7f8218071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45754/0 (socket says 172.21.15.77:45754) 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f8216ffd700 1 -- 172.21.15.77:0/233797892 learned_addr learned my addr 172.21.15.77:0/233797892 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f8216ffd700 1 -- 172.21.15.77:0/233797892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8208005040 con 0x7f82180fcbd0 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f8216ffd700 1 --2- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 0x7f8218071e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8208009680 tx=0x7f8208009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f820f7fe700 1 -- 172.21.15.77:0/233797892 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f820800bed0 con 0x7f82180fcbd0 2024-02-19T18:51:11.139 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f820f7fe700 1 -- 172.21.15.77:0/233797892 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f820800b5c0 con 0x7f82180fcbd0 2024-02-19T18:51:11.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f820f7fe700 1 -- 172.21.15.77:0/233797892 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f820801c5b0 con 0x7f82180fcbd0 2024-02-19T18:51:11.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82180704e0 con 0x7f82180fcbd0 2024-02-19T18:51:11.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.122+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82180709d0 con 0x7f82180fcbd0 2024-02-19T18:51:11.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.123+0000 7f820f7fe700 1 -- 172.21.15.77:0/233797892 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f8208004d30 con 0x7f82180fcbd0 2024-02-19T18:51:11.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.123+0000 7f820f7fe700 1 --2- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8204036870 0x7f8204038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:11.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.123+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82181069a0 con 0x7f82180fcbd0 2024-02-19T18:51:11.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.123+0000 7f820f7fe700 1 -- 172.21.15.77:0/233797892 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8208049a90 con 0x7f82180fcbd0 2024-02-19T18:51:11.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.123+0000 7f820ffff700 1 --2- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8204036870 0x7f8204038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:11.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.124+0000 7f820ffff700 1 --2- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8204036870 0x7f8204038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8200009990 tx=0x7f8200006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:11.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.128+0000 7f820f7fe700 1 -- 172.21.15.77:0/233797892 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8208011940 con 0x7f82180fcbd0 2024-02-19T18:51:11.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.321+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 --> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f821802b3b0 con 0x7f8204036870 2024-02-19T18:51:11.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.323+0000 7f820f7fe700 1 -- 172.21.15.77:0/233797892 <== mgr.14162 v2:172.21.15.77:6800/3287530536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f821802b3b0 con 0x7f8204036870 2024-02-19T18:51:11.338 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:51:11.338 INFO:teuthology.orchestra.run.smithi077.stdout:[{"addr": "172.21.15.77", "hostname": "smithi077", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2024-02-19T18:51:11.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8204036870 msgr2=0x7f8204038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:11.340 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 --2- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8204036870 0x7f8204038d20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8200009990 tx=0x7f8200006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 msgr2=0x7f8218071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 --2- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 0x7f8218071e60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8208009680 tx=0x7f8208009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 shutdown_connections 2024-02-19T18:51:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 --2- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8204036870 0x7f8204038d20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 --2- 172.21.15.77:0/233797892 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82180fcbd0 0x7f8218071e60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 >> 172.21.15.77:0/233797892 conn(0x7f82180f8290 msgr2=0x7f82180f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 shutdown_connections 2024-02-19T18:51:11.341 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:11.325+0000 7f821d53d700 1 -- 172.21.15.77:0/233797892 wait complete. 2024-02-19T18:51:11.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:11 smithi077 ceph-mon[29247]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:11.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:11 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:11.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:11 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:11.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:11 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:11.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:11 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:11.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:11 smithi077 ceph-mon[29247]: Deploying daemon alertmanager.smithi077 on smithi077 2024-02-19T18:51:11.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:11 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:11.936 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-19T18:51:11.936 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd crush tunables default 2024-02-19T18:51:12.185 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:51:12.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:12 smithi077 ceph-mon[29247]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:51:13.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:13 smithi077 ceph-mon[29247]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:14.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 -- 172.21.15.77:0/1302929072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 msgr2=0x7f17140fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:14.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 --2- 172.21.15.77:0/1302929072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 0x7f17140fbee0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1710009c60 tx=0x7f1710004d10 comp rx=0 tx=0).stop 2024-02-19T18:51:14.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 -- 172.21.15.77:0/1302929072 shutdown_connections 2024-02-19T18:51:14.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 --2- 172.21.15.77:0/1302929072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 0x7f17140fbee0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:14.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 -- 172.21.15.77:0/1302929072 >> 172.21.15.77:0/1302929072 conn(0x7f17140f7190 msgr2=0x7f17140f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:14.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 -- 172.21.15.77:0/1302929072 shutdown_connections 2024-02-19T18:51:14.134 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 -- 172.21.15.77:0/1302929072 wait complete. 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 Processor -- start 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.111+0000 7f171b2cd700 1 -- start start 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f171b2cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 0x7f171410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f171b2cd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171410d400 con 0x7f17140fbad0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f1719069700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 0x7f171410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f1719069700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 0x7f171410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:45774/0 (socket says 172.21.15.77:45774) 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f1719069700 1 -- 172.21.15.77:0/3580630782 learned_addr learned my addr 172.21.15.77:0/3580630782 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f1719069700 1 -- 172.21.15.77:0/3580630782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1710009940 con 0x7f17140fbad0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f1719069700 1 --2- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 0x7f171410cec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1710015040 tx=0x7f1710004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f1709ffb700 1 -- 172.21.15.77:0/3580630782 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1710007c80 con 0x7f17140fbad0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f1709ffb700 1 -- 172.21.15.77:0/3580630782 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f171001d910 con 0x7f17140fbad0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f1709ffb700 1 -- 172.21.15.77:0/3580630782 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17100129f0 con 0x7f17140fbad0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171410d600 con 0x7f17140fbad0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.112+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171410e2a0 con 0x7f17140fbad0 2024-02-19T18:51:14.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.113+0000 7f1709ffb700 1 -- 172.21.15.77:0/3580630782 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f1710019040 con 0x7f17140fbad0 2024-02-19T18:51:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.113+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171405f5c0 con 0x7f17140fbad0 2024-02-19T18:51:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.114+0000 7f1709ffb700 1 --2- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f17000368c0 0x7f1700038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.114+0000 7f1709ffb700 1 -- 172.21.15.77:0/3580630782 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f171004a390 con 0x7f17140fbad0 2024-02-19T18:51:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.114+0000 7f1718868700 1 --2- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f17000368c0 0x7f1700038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.114+0000 7f1718868700 1 --2- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f17000368c0 0x7f1700038d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1704009990 tx=0x7f1704006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:14.136 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.118+0000 7f1709ffb700 1 -- 172.21.15.77:0/3580630782 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f171002eb40 con 0x7f17140fbad0 2024-02-19T18:51:14.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:14.317+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f171410ed90 con 0x7f17140fbad0 2024-02-19T18:51:15.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.257+0000 7f1709ffb700 1 -- 172.21.15.77:0/3580630782 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1710017020 con 0x7f17140fbad0 2024-02-19T18:51:15.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f17000368c0 msgr2=0x7f1700038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:15.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 --2- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f17000368c0 0x7f1700038d70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1704009990 tx=0x7f1704006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:15.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 msgr2=0x7f171410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:15.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 --2- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 0x7f171410cec0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1710015040 tx=0x7f1710004d10 comp rx=0 tx=0).stop 2024-02-19T18:51:15.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 shutdown_connections 2024-02-19T18:51:15.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 --2- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f17000368c0 0x7f1700038d70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:15.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 --2- 172.21.15.77:0/3580630782 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17140fbad0 0x7f171410cec0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:15.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 >> 172.21.15.77:0/3580630782 conn(0x7f17140f7190 msgr2=0x7f1714065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:15.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.259+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 shutdown_connections 2024-02-19T18:51:15.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:51:15.260+0000 7f171b2cd700 1 -- 172.21.15.77:0/3580630782 wait complete. 2024-02-19T18:51:15.276 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-02-19T18:51:15.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:15 smithi077 ceph-mon[29247]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:15.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:15 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3580630782' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-19T18:51:16.048 INFO:tasks.cephadm:Adding mon.smithi077 on smithi077 2024-02-19T18:51:16.048 INFO:tasks.cephadm:Adding mon.smithi104 on smithi104 2024-02-19T18:51:16.049 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch apply mon '2;smithi077:172.21.15.77=smithi077;smithi104:172.21.15.104=smithi104' 2024-02-19T18:51:16.298 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:16.344 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:16.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:16 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3580630782' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-19T18:51:16.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:16 smithi077 ceph-mon[29247]: osdmap e4: 0 total, 0 up, 0 in 2024-02-19T18:51:17.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:17 smithi077 ceph-mon[29247]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:17.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.813+0000 7f2844267700 1 -- 172.21.15.104:0/3922993579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 msgr2=0x7f283c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.813+0000 7f2844267700 1 --2- 172.21.15.104:0/3922993579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 0x7f283c0fcfe0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2838009c60 tx=0x7f2838004d10 comp rx=0 tx=0).stop 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.813+0000 7f2844267700 1 -- 172.21.15.104:0/3922993579 shutdown_connections 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.813+0000 7f2844267700 1 --2- 172.21.15.104:0/3922993579 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 0x7f283c0fcfe0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.813+0000 7f2844267700 1 -- 172.21.15.104:0/3922993579 >> 172.21.15.104:0/3922993579 conn(0x7f283c0f8270 msgr2=0x7f283c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.813+0000 7f2844267700 1 -- 172.21.15.104:0/3922993579 shutdown_connections 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.813+0000 7f2844267700 1 -- 172.21.15.104:0/3922993579 wait complete. 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.814+0000 7f2844267700 1 Processor -- start 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.814+0000 7f2844267700 1 -- start start 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.814+0000 7f2844267700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 0x7f283c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.814+0000 7f2844267700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c10d430 con 0x7f283c0fcbd0 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.814+0000 7f2842003700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 0x7f283c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.814+0000 7f2842003700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 0x7f283c10cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:41548/0 (socket says 172.21.15.104:41548) 2024-02-19T18:51:17.831 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.814+0000 7f2842003700 1 -- 172.21.15.104:0/3641508653 learned_addr learned my addr 172.21.15.104:0/3641508653 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:17.832 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.815+0000 7f2842003700 1 -- 172.21.15.104:0/3641508653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2838009940 con 0x7f283c0fcbd0 2024-02-19T18:51:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.815+0000 7f2842003700 1 --2- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 0x7f283c10cef0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f28380097d0 tx=0x7f2838004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.815+0000 7f2832ffd700 1 -- 172.21.15.104:0/3641508653 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f283801d440 con 0x7f283c0fcbd0 2024-02-19T18:51:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.815+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c10d630 con 0x7f283c0fcbd0 2024-02-19T18:51:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.815+0000 7f2832ffd700 1 -- 172.21.15.104:0/3641508653 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f283801da70 con 0x7f283c0fcbd0 2024-02-19T18:51:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.815+0000 7f2832ffd700 1 -- 172.21.15.104:0/3641508653 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2838012a10 con 0x7f283c0fcbd0 2024-02-19T18:51:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.815+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c10e2d0 con 0x7f283c0fcbd0 2024-02-19T18:51:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.816+0000 7f2832ffd700 1 -- 172.21.15.104:0/3641508653 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f2838019040 con 0x7f283c0fcbd0 2024-02-19T18:51:17.833 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.816+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c05f5c0 con 0x7f283c0fcbd0 2024-02-19T18:51:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.816+0000 7f2832ffd700 1 --2- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2828036870 0x7f2828038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.816+0000 7f2832ffd700 1 -- 172.21.15.104:0/3641508653 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f283804a460 con 0x7f283c0fcbd0 2024-02-19T18:51:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.817+0000 7f2841802700 1 --2- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2828036870 0x7f2828038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:17.834 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.817+0000 7f2841802700 1 --2- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2828036870 0x7f2828038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f282c009990 tx=0x7f282c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:17.837 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:17.821+0000 7f2832ffd700 1 -- 172.21.15.104:0/3641508653 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2838024310 con 0x7f283c0fcbd0 2024-02-19T18:51:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.009+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 --> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi077:172.21.15.77=smithi077;smithi104:172.21.15.104=smithi104", "target": ["mon-mgr", ""]}) v1 -- 0x7f283c10eee0 con 0x7f2828036870 2024-02-19T18:51:18.031 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.015+0000 7f2832ffd700 1 -- 172.21.15.104:0/3641508653 <== mgr.14162 v2:172.21.15.77:6800/3287530536 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f283c10eee0 con 0x7f2828036870 2024-02-19T18:51:18.032 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mon update... 2024-02-19T18:51:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2828036870 msgr2=0x7f2828038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:18.033 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 --2- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2828036870 0x7f2828038d20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f282c009990 tx=0x7f282c006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 msgr2=0x7f283c10cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 --2- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 0x7f283c10cef0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f28380097d0 tx=0x7f2838004d10 comp rx=0 tx=0).stop 2024-02-19T18:51:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 shutdown_connections 2024-02-19T18:51:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 --2- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2828036870 0x7f2828038d20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 --2- 172.21.15.104:0/3641508653 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f283c0fcbd0 0x7f283c10cef0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 >> 172.21.15.104:0/3641508653 conn(0x7f283c0f8270 msgr2=0x7f283c0f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 shutdown_connections 2024-02-19T18:51:18.034 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:18.017+0000 7f2844267700 1 -- 172.21.15.104:0/3641508653 wait complete. 2024-02-19T18:51:18.555 DEBUG:teuthology.orchestra.run.smithi104:mon.smithi104> sudo journalctl -f -n 0 -u ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi104.service 2024-02-19T18:51:18.558 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:18.558 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:18.834 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:18.882 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:19.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:19 smithi077 ceph-mon[29247]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi077:172.21.15.77=smithi077;smithi104:172.21.15.104=smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:51:19.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:19 smithi077 ceph-mon[29247]: Saving service mon spec with placement smithi077:172.21.15.77=smithi077;smithi104:172.21.15.104=smithi104;count:2 2024-02-19T18:51:19.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:19 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:19.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:19 smithi077 ceph-mon[29247]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:20.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 -- 172.21.15.104:0/3388064353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 msgr2=0x7f08b00fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:20.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 --2- 172.21.15.104:0/3388064353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 0x7f08b00fbed0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f08a00048c0 tx=0x7f08a0009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:20.246 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 -- 172.21.15.104:0/3388064353 shutdown_connections 2024-02-19T18:51:20.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 --2- 172.21.15.104:0/3388064353 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 0x7f08b00fbed0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:20.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 -- 172.21.15.104:0/3388064353 >> 172.21.15.104:0/3388064353 conn(0x7f08b00f71a0 msgr2=0x7f08b00f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:20.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 -- 172.21.15.104:0/3388064353 shutdown_connections 2024-02-19T18:51:20.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 -- 172.21.15.104:0/3388064353 wait complete. 2024-02-19T18:51:20.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 Processor -- start 2024-02-19T18:51:20.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.230+0000 7f08b83d9700 1 -- start start 2024-02-19T18:51:20.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.231+0000 7f08b83d9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 0x7f08b010be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:20.247 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.231+0000 7f08b83d9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b010c390 con 0x7f08b00fbac0 2024-02-19T18:51:20.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.231+0000 7f08b6175700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 0x7f08b010be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:20.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.231+0000 7f08b6175700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 0x7f08b010be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:41556/0 (socket says 172.21.15.104:41556) 2024-02-19T18:51:20.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.231+0000 7f08b6175700 1 -- 172.21.15.104:0/3719424416 learned_addr learned my addr 172.21.15.104:0/3719424416 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:20.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.231+0000 7f08b6175700 1 -- 172.21.15.104:0/3719424416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a0005040 con 0x7f08b00fbac0 2024-02-19T18:51:20.248 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.231+0000 7f08b6175700 1 --2- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 0x7f08b010be50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009680 tx=0x7f08a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:20.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.232+0000 7f08a6ffd700 1 -- 172.21.15.104:0/3719424416 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08a000b5c0 con 0x7f08b00fbac0 2024-02-19T18:51:20.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.232+0000 7f08a6ffd700 1 -- 172.21.15.104:0/3719424416 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f08a0004e60 con 0x7f08b00fbac0 2024-02-19T18:51:20.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.232+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b010c590 con 0x7f08b00fbac0 2024-02-19T18:51:20.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.232+0000 7f08a6ffd700 1 -- 172.21.15.104:0/3719424416 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08a001e7a0 con 0x7f08b00fbac0 2024-02-19T18:51:20.249 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.232+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b010d150 con 0x7f08b00fbac0 2024-02-19T18:51:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.233+0000 7f08a6ffd700 1 -- 172.21.15.104:0/3719424416 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f08a001ea60 con 0x7f08b00fbac0 2024-02-19T18:51:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.233+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b005f5c0 con 0x7f08b00fbac0 2024-02-19T18:51:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.233+0000 7f08a6ffd700 1 --2- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f089c036870 0x7f089c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.233+0000 7f08a6ffd700 1 -- 172.21.15.104:0/3719424416 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f08a004d5e0 con 0x7f08b00fbac0 2024-02-19T18:51:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.233+0000 7f08b5974700 1 --2- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f089c036870 0x7f089c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:20.250 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.233+0000 7f08b5974700 1 --2- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f089c036870 0x7f089c038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009990 tx=0x7f08ac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:20.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.238+0000 7f08a6ffd700 1 -- 172.21.15.104:0/3719424416 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f08a00273c0 con 0x7f08b00fbac0 2024-02-19T18:51:20.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.493+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08b004bed0 con 0x7f08b00fbac0 2024-02-19T18:51:20.509 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.493+0000 7f08a6ffd700 1 -- 172.21.15.104:0/3719424416 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f08a0011370 con 0x7f08b00fbac0 2024-02-19T18:51:20.509 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:20.510 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:20.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f089c036870 msgr2=0x7f089c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 --2- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f089c036870 0x7f089c038d20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009990 tx=0x7f08ac006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 msgr2=0x7f08b010be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 --2- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 0x7f08b010be50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f08a0009680 tx=0x7f08a0009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 shutdown_connections 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 --2- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f089c036870 0x7f089c038d20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 --2- 172.21.15.104:0/3719424416 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f08b00fbac0 0x7f08b010be50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 >> 172.21.15.104:0/3719424416 conn(0x7f08b00f71a0 msgr2=0x7f08b00f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 shutdown_connections 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:20.495+0000 7f08b83d9700 1 -- 172.21.15.104:0/3719424416 wait complete. 2024-02-19T18:51:20.512 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:21.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:21 smithi077 ceph-mon[29247]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:21.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:21 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3719424416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:22.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:22.018 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:22.264 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:22.311 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:23.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:23 smithi077 ceph-mon[29247]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:23.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.641+0000 7f77f8de0700 1 -- 172.21.15.104:0/3872529107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 msgr2=0x7f77f40fe610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:23.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.641+0000 7f77f8de0700 1 --2- 172.21.15.104:0/3872529107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 0x7f77f40fe610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f77e40048c0 tx=0x7f77e4009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:23.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.641+0000 7f77f8de0700 1 -- 172.21.15.104:0/3872529107 shutdown_connections 2024-02-19T18:51:23.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.641+0000 7f77f8de0700 1 --2- 172.21.15.104:0/3872529107 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 0x7f77f40fe610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:23.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.641+0000 7f77f8de0700 1 -- 172.21.15.104:0/3872529107 >> 172.21.15.104:0/3872529107 conn(0x7f77f40f69c0 msgr2=0x7f77f40f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:23.658 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.641+0000 7f77f8de0700 1 -- 172.21.15.104:0/3872529107 shutdown_connections 2024-02-19T18:51:23.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.641+0000 7f77f8de0700 1 -- 172.21.15.104:0/3872529107 wait complete. 2024-02-19T18:51:23.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.642+0000 7f77f8de0700 1 Processor -- start 2024-02-19T18:51:23.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.642+0000 7f77f8de0700 1 -- start start 2024-02-19T18:51:23.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.642+0000 7f77f8de0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 0x7f77f410cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:23.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.642+0000 7f77f8de0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f410d460 con 0x7f77f40fe200 2024-02-19T18:51:23.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.642+0000 7f77f259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 0x7f77f410cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:23.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.642+0000 7f77f259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 0x7f77f410cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:41586/0 (socket says 172.21.15.104:41586) 2024-02-19T18:51:23.659 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.642+0000 7f77f259c700 1 -- 172.21.15.104:0/1153446619 learned_addr learned my addr 172.21.15.104:0/1153446619 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:23.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.643+0000 7f77f259c700 1 -- 172.21.15.104:0/1153446619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e4005040 con 0x7f77f40fe200 2024-02-19T18:51:23.660 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.643+0000 7f77f259c700 1 --2- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 0x7f77f410cf20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009300 tx=0x7f77e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.643+0000 7f77e2ffd700 1 -- 172.21.15.104:0/1153446619 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77e4004e60 con 0x7f77f40fe200 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.643+0000 7f77e2ffd700 1 -- 172.21.15.104:0/1153446619 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f77e4009680 con 0x7f77f40fe200 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.643+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f410d660 con 0x7f77f40fe200 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.643+0000 7f77e2ffd700 1 -- 172.21.15.104:0/1153446619 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f77e401c5b0 con 0x7f77f40fe200 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.643+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f410e300 con 0x7f77f40fe200 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.644+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f405f5c0 con 0x7f77f40fe200 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.644+0000 7f77e2ffd700 1 -- 172.21.15.104:0/1153446619 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f77e40097f0 con 0x7f77f40fe200 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.644+0000 7f77e2ffd700 1 --2- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f77dc036870 0x7f77dc038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.644+0000 7f77e2ffd700 1 -- 172.21.15.104:0/1153446619 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f77e4049b00 con 0x7f77f40fe200 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.644+0000 7f77f1d9b700 1 --2- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f77dc036870 0x7f77dc038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:23.661 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.645+0000 7f77f1d9b700 1 --2- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f77dc036870 0x7f77dc038d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009990 tx=0x7f77e8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:23.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.649+0000 7f77e2ffd700 1 -- 172.21.15.104:0/1153446619 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f77e4011550 con 0x7f77f40fe200 2024-02-19T18:51:23.922 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.906+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77f410ee50 con 0x7f77f40fe200 2024-02-19T18:51:23.923 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.907+0000 7f77e2ffd700 1 -- 172.21.15.104:0/1153446619 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f77e4011370 con 0x7f77f40fe200 2024-02-19T18:51:23.923 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:23.923 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f77dc036870 msgr2=0x7f77dc038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 --2- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f77dc036870 0x7f77dc038d20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f77e8009990 tx=0x7f77e8006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 msgr2=0x7f77f410cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 --2- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 0x7f77f410cf20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f77e4009300 tx=0x7f77e4009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:23.925 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 shutdown_connections 2024-02-19T18:51:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 --2- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f77dc036870 0x7f77dc038d20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 --2- 172.21.15.104:0/1153446619 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f77f40fe200 0x7f77f410cf20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 >> 172.21.15.104:0/1153446619 conn(0x7f77f40f69c0 msgr2=0x7f77f40f7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 shutdown_connections 2024-02-19T18:51:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:23.909+0000 7f77f8de0700 1 -- 172.21.15.104:0/1153446619 wait complete. 2024-02-19T18:51:23.926 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:24.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:24 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1153446619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:25.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:25 smithi077 ceph-mon[29247]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:25.654 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:25.655 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:25.902 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:25.948 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:27.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.400+0000 7fc385782700 1 -- 172.21.15.104:0/514553876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 msgr2=0x7fc3800fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:27.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.400+0000 7fc385782700 1 --2- 172.21.15.104:0/514553876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 0x7fc3800fbed0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc3700048c0 tx=0x7fc370009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:27.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.400+0000 7fc385782700 1 -- 172.21.15.104:0/514553876 shutdown_connections 2024-02-19T18:51:27.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.400+0000 7fc385782700 1 --2- 172.21.15.104:0/514553876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 0x7fc3800fbed0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:27.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.400+0000 7fc385782700 1 -- 172.21.15.104:0/514553876 >> 172.21.15.104:0/514553876 conn(0x7fc3800f71a0 msgr2=0x7fc3800f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:27.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.400+0000 7fc385782700 1 -- 172.21.15.104:0/514553876 shutdown_connections 2024-02-19T18:51:27.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.400+0000 7fc385782700 1 -- 172.21.15.104:0/514553876 wait complete. 2024-02-19T18:51:27.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.401+0000 7fc385782700 1 Processor -- start 2024-02-19T18:51:27.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.401+0000 7fc385782700 1 -- start start 2024-02-19T18:51:27.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.401+0000 7fc385782700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 0x7fc38010fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:27.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.401+0000 7fc385782700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc380066850 con 0x7fc3800fbac0 2024-02-19T18:51:27.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.401+0000 7fc37effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 0x7fc38010fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.401+0000 7fc37effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 0x7fc38010fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:39446/0 (socket says 172.21.15.104:39446) 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.401+0000 7fc37effd700 1 -- 172.21.15.104:0/1611627502 learned_addr learned my addr 172.21.15.104:0/1611627502 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.402+0000 7fc37effd700 1 -- 172.21.15.104:0/1611627502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc370005040 con 0x7fc3800fbac0 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.402+0000 7fc37effd700 1 --2- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 0x7fc38010fb90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc370009680 tx=0x7fc370009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.402+0000 7fc3677fe700 1 -- 172.21.15.104:0/1611627502 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc37000bed0 con 0x7fc3800fbac0 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.402+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc380066a50 con 0x7fc3800fbac0 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.402+0000 7fc3677fe700 1 -- 172.21.15.104:0/1611627502 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc37000b5c0 con 0x7fc3800fbac0 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.402+0000 7fc3677fe700 1 -- 172.21.15.104:0/1611627502 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc37001c5b0 con 0x7fc3800fbac0 2024-02-19T18:51:27.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.402+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc380110310 con 0x7fc3800fbac0 2024-02-19T18:51:27.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.403+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc380105890 con 0x7fc3800fbac0 2024-02-19T18:51:27.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.403+0000 7fc3677fe700 1 -- 172.21.15.104:0/1611627502 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc370013a80 con 0x7fc3800fbac0 2024-02-19T18:51:27.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.403+0000 7fc3677fe700 1 --2- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc368036820 0x7fc368038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:27.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.403+0000 7fc3677fe700 1 -- 172.21.15.104:0/1611627502 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc370049ab0 con 0x7fc3800fbac0 2024-02-19T18:51:27.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.403+0000 7fc37e7fc700 1 --2- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc368036820 0x7fc368038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:27.420 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.404+0000 7fc37e7fc700 1 --2- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc368036820 0x7fc368038cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc374009990 tx=0x7fc374006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:27.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.408+0000 7fc3677fe700 1 -- 172.21.15.104:0/1611627502 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc370011940 con 0x7fc3800fbac0 2024-02-19T18:51:27.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:27 smithi077 ceph-mon[29247]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:27.678 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.662+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc380101880 con 0x7fc3800fbac0 2024-02-19T18:51:27.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.663+0000 7fc3677fe700 1 -- 172.21.15.104:0/1611627502 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc370011370 con 0x7fc3800fbac0 2024-02-19T18:51:27.679 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:27.679 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:27.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc368036820 msgr2=0x7fc368038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:27.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 --2- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc368036820 0x7fc368038cd0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc374009990 tx=0x7fc374006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:27.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 msgr2=0x7fc38010fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:27.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 --2- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 0x7fc38010fb90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc370009680 tx=0x7fc370009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:27.681 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 shutdown_connections 2024-02-19T18:51:27.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 --2- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc368036820 0x7fc368038cd0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:27.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 --2- 172.21.15.104:0/1611627502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc3800fbac0 0x7fc38010fb90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:27.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 >> 172.21.15.104:0/1611627502 conn(0x7fc3800f71a0 msgr2=0x7fc3800f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:27.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 shutdown_connections 2024-02-19T18:51:27.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:27.665+0000 7fc385782700 1 -- 172.21.15.104:0/1611627502 wait complete. 2024-02-19T18:51:27.682 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:28.295 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:28 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1611627502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:29.093 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:29.093 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:29.344 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:29.390 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:29.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:29.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:29.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:29.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:29.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:29.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:29.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:29.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T18:51:29.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:29 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:30.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.712+0000 7fcc12431700 1 -- 172.21.15.104:0/3558159953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 msgr2=0x7fcc0c0fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:30.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.712+0000 7fcc12431700 1 --2- 172.21.15.104:0/3558159953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 0x7fcc0c0fbee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc0048c0 tx=0x7fcbfc009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:30.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.713+0000 7fcc12431700 1 -- 172.21.15.104:0/3558159953 shutdown_connections 2024-02-19T18:51:30.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.713+0000 7fcc12431700 1 --2- 172.21.15.104:0/3558159953 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 0x7fcc0c0fbee0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:30.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.713+0000 7fcc12431700 1 -- 172.21.15.104:0/3558159953 >> 172.21.15.104:0/3558159953 conn(0x7fcc0c0f7190 msgr2=0x7fcc0c0f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:30.729 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.713+0000 7fcc12431700 1 -- 172.21.15.104:0/3558159953 shutdown_connections 2024-02-19T18:51:30.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.713+0000 7fcc12431700 1 -- 172.21.15.104:0/3558159953 wait complete. 2024-02-19T18:51:30.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.713+0000 7fcc12431700 1 Processor -- start 2024-02-19T18:51:30.730 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.713+0000 7fcc12431700 1 -- start start 2024-02-19T18:51:30.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc12431700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 0x7fcc0c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:30.731 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc12431700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c10d400 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc0bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 0x7fcc0c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc0bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 0x7fcc0c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:39470/0 (socket says 172.21.15.104:39470) 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc0bfff700 1 -- 172.21.15.104:0/1376033267 learned_addr learned my addr 172.21.15.104:0/1376033267 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc0bfff700 1 -- 172.21.15.104:0/1376033267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbfc005040 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc0bfff700 1 --2- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 0x7fcc0c10cec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009680 tx=0x7fcbfc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc08ff9700 1 -- 172.21.15.104:0/1376033267 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbfc00bed0 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc08ff9700 1 -- 172.21.15.104:0/1376033267 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbfc00b5c0 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc08ff9700 1 -- 172.21.15.104:0/1376033267 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbfc01c5b0 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c10d600 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.714+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c10e2a0 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.715+0000 7fcc08ff9700 1 -- 172.21.15.104:0/1376033267 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fcbfc01c820 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.715+0000 7fcc08ff9700 1 --2- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fcbf4036820 0x7fcbf4038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.715+0000 7fcc08ff9700 1 -- 172.21.15.104:0/1376033267 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcbfc049b80 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.732 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.715+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c05f5c0 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.715+0000 7fcc0b7fe700 1 --2- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fcbf4036820 0x7fcbf4038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:30.733 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.716+0000 7fcc0b7fe700 1 --2- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fcbf4036820 0x7fcbf4038cd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009990 tx=0x7fcc00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:30.736 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.720+0000 7fcc08ff9700 1 -- 172.21.15.104:0/1376033267 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcbfc011dd0 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:30 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-19T18:51:30.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:30 smithi077 ceph-mon[29247]: Deploying daemon grafana.smithi077 on smithi077 2024-02-19T18:51:30.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.978+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc0c10ed50 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.995 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.979+0000 7fcc08ff9700 1 -- 172.21.15.104:0/1376033267 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fcbfc011370 con 0x7fcc0c0fbad0 2024-02-19T18:51:30.995 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:30.995 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:30.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.980+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fcbf4036820 msgr2=0x7fcbf4038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.980+0000 7fcc12431700 1 --2- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fcbf4036820 0x7fcbf4038cd0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcc00009990 tx=0x7fcc00006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.980+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 msgr2=0x7fcc0c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.980+0000 7fcc12431700 1 --2- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 0x7fcc0c10cec0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009680 tx=0x7fcbfc009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.980+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 shutdown_connections 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.980+0000 7fcc12431700 1 --2- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fcbf4036820 0x7fcbf4038cd0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.980+0000 7fcc12431700 1 --2- 172.21.15.104:0/1376033267 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc0c0fbad0 0x7fcc0c10cec0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.980+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 >> 172.21.15.104:0/1376033267 conn(0x7fcc0c0f7190 msgr2=0x7fcc0c065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.981+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 shutdown_connections 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:30.981+0000 7fcc12431700 1 -- 172.21.15.104:0/1376033267 wait complete. 2024-02-19T18:51:30.997 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:31.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:31 smithi077 ceph-mon[29247]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:31.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:31 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1376033267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:31.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:31 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:51:32.614 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:32.614 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:32.861 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:32.908 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:33.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:33 smithi077 ceph-mon[29247]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.300+0000 7ffa3033e700 1 -- 172.21.15.104:0/1519265852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 msgr2=0x7ffa280fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.300+0000 7ffa3033e700 1 --2- 172.21.15.104:0/1519265852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 0x7ffa280fcfe0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ffa180048c0 tx=0x7ffa18009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.300+0000 7ffa3033e700 1 -- 172.21.15.104:0/1519265852 shutdown_connections 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.300+0000 7ffa3033e700 1 --2- 172.21.15.104:0/1519265852 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 0x7ffa280fcfe0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.300+0000 7ffa3033e700 1 -- 172.21.15.104:0/1519265852 >> 172.21.15.104:0/1519265852 conn(0x7ffa280f8270 msgr2=0x7ffa280fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.300+0000 7ffa3033e700 1 -- 172.21.15.104:0/1519265852 shutdown_connections 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.300+0000 7ffa3033e700 1 -- 172.21.15.104:0/1519265852 wait complete. 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.301+0000 7ffa3033e700 1 Processor -- start 2024-02-19T18:51:34.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.301+0000 7ffa3033e700 1 -- start start 2024-02-19T18:51:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.301+0000 7ffa3033e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 0x7ffa2810cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.301+0000 7ffa3033e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2810d430 con 0x7ffa280fcbd0 2024-02-19T18:51:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.301+0000 7ffa2e0da700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 0x7ffa2810cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.301+0000 7ffa2e0da700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 0x7ffa2810cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:39488/0 (socket says 172.21.15.104:39488) 2024-02-19T18:51:34.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.301+0000 7ffa2e0da700 1 -- 172.21.15.104:0/2290806146 learned_addr learned my addr 172.21.15.104:0/2290806146 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.301+0000 7ffa2e0da700 1 -- 172.21.15.104:0/2290806146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa18005040 con 0x7ffa280fcbd0 2024-02-19T18:51:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.302+0000 7ffa2e0da700 1 --2- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 0x7ffa2810cef0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009b80 tx=0x7ffa18009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.302+0000 7ffa1effd700 1 -- 172.21.15.104:0/2290806146 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1800b5c0 con 0x7ffa280fcbd0 2024-02-19T18:51:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.302+0000 7ffa1effd700 1 -- 172.21.15.104:0/2290806146 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ffa1800be70 con 0x7ffa280fcbd0 2024-02-19T18:51:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.302+0000 7ffa1effd700 1 -- 172.21.15.104:0/2290806146 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa1801c660 con 0x7ffa280fcbd0 2024-02-19T18:51:34.319 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.302+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2810d630 con 0x7ffa280fcbd0 2024-02-19T18:51:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.302+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2810e2d0 con 0x7ffa280fcbd0 2024-02-19T18:51:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.303+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2805f5c0 con 0x7ffa280fcbd0 2024-02-19T18:51:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.303+0000 7ffa1effd700 1 -- 172.21.15.104:0/2290806146 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ffa18004d30 con 0x7ffa280fcbd0 2024-02-19T18:51:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.303+0000 7ffa1effd700 1 --2- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ffa140368c0 0x7ffa14038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.303+0000 7ffa1effd700 1 -- 172.21.15.104:0/2290806146 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ffa180499a0 con 0x7ffa280fcbd0 2024-02-19T18:51:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.303+0000 7ffa2d8d9700 1 --2- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ffa140368c0 0x7ffa14038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:34.320 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.304+0000 7ffa2d8d9700 1 --2- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ffa140368c0 0x7ffa14038d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009990 tx=0x7ffa24006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:34.324 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.308+0000 7ffa1effd700 1 -- 172.21.15.104:0/2290806146 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ffa18025400 con 0x7ffa280fcbd0 2024-02-19T18:51:34.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.562+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa2810ee40 con 0x7ffa280fcbd0 2024-02-19T18:51:34.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.563+0000 7ffa1effd700 1 -- 172.21.15.104:0/2290806146 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ffa18023300 con 0x7ffa280fcbd0 2024-02-19T18:51:34.579 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:34.579 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.564+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ffa140368c0 msgr2=0x7ffa14038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.564+0000 7ffa3033e700 1 --2- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ffa140368c0 0x7ffa14038d70 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009990 tx=0x7ffa24006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.564+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 msgr2=0x7ffa2810cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.564+0000 7ffa3033e700 1 --2- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 0x7ffa2810cef0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009b80 tx=0x7ffa18009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.564+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 shutdown_connections 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.564+0000 7ffa3033e700 1 --2- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ffa140368c0 0x7ffa14038d70 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.564+0000 7ffa3033e700 1 --2- 172.21.15.104:0/2290806146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ffa280fcbd0 0x7ffa2810cef0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.564+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 >> 172.21.15.104:0/2290806146 conn(0x7ffa280f8270 msgr2=0x7ffa280f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.565+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 shutdown_connections 2024-02-19T18:51:34.581 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:34.565+0000 7ffa3033e700 1 -- 172.21.15.104:0/2290806146 wait complete. 2024-02-19T18:51:34.582 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:35.806 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:35 smithi077 ceph-mon[29247]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:35.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:35 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2290806146' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:36.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:36.174 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:36.422 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:36.468 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:37.812 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.795+0000 7fc40f328700 1 -- 172.21.15.104:0/422800458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 msgr2=0x7fc4080fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.795+0000 7fc40f328700 1 --2- 172.21.15.104:0/422800458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 0x7fc4080fbee0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc3f80048c0 tx=0x7fc3f8009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.795+0000 7fc40f328700 1 -- 172.21.15.104:0/422800458 shutdown_connections 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.795+0000 7fc40f328700 1 --2- 172.21.15.104:0/422800458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 0x7fc4080fbee0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.795+0000 7fc40f328700 1 -- 172.21.15.104:0/422800458 >> 172.21.15.104:0/422800458 conn(0x7fc4080f7190 msgr2=0x7fc4080f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.795+0000 7fc40f328700 1 -- 172.21.15.104:0/422800458 shutdown_connections 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.795+0000 7fc40f328700 1 -- 172.21.15.104:0/422800458 wait complete. 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.796+0000 7fc40f328700 1 Processor -- start 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.796+0000 7fc40f328700 1 -- start start 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.796+0000 7fc40f328700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 0x7fc40810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.796+0000 7fc40f328700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40810d400 con 0x7fc4080fbad0 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.796+0000 7fc40d0c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 0x7fc40810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.796+0000 7fc40d0c4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 0x7fc40810cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:45892/0 (socket says 172.21.15.104:45892) 2024-02-19T18:51:37.813 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.796+0000 7fc40d0c4700 1 -- 172.21.15.104:0/1150390449 learned_addr learned my addr 172.21.15.104:0/1150390449 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.796+0000 7fc40d0c4700 1 -- 172.21.15.104:0/1150390449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f8005040 con 0x7fc4080fbad0 2024-02-19T18:51:37.814 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.797+0000 7fc40d0c4700 1 --2- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 0x7fc40810cec0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009680 tx=0x7fc3f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.797+0000 7fc3fdffb700 1 -- 172.21.15.104:0/1150390449 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3f800bed0 con 0x7fc4080fbad0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.797+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40810d600 con 0x7fc4080fbad0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.797+0000 7fc3fdffb700 1 -- 172.21.15.104:0/1150390449 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc3f800b5c0 con 0x7fc4080fbad0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.797+0000 7fc3fdffb700 1 -- 172.21.15.104:0/1150390449 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3f801c5b0 con 0x7fc4080fbad0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.797+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40810e2a0 con 0x7fc4080fbad0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.798+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40805f5c0 con 0x7fc4080fbad0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.798+0000 7fc3fdffb700 1 -- 172.21.15.104:0/1150390449 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fc3f8013a80 con 0x7fc4080fbad0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.798+0000 7fc3fdffb700 1 --2- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc3f4036820 0x7fc3f4038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.798+0000 7fc3fdffb700 1 -- 172.21.15.104:0/1150390449 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc3f8049ab0 con 0x7fc4080fbad0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.798+0000 7fc40c8c3700 1 --2- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc3f4036820 0x7fc3f4038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:37.815 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.799+0000 7fc40c8c3700 1 --2- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc3f4036820 0x7fc3f4038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc404009990 tx=0x7fc404006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:37.819 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:37.803+0000 7fc3fdffb700 1 -- 172.21.15.104:0/1150390449 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc3f8011940 con 0x7fc4080fbad0 2024-02-19T18:51:38.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:37 smithi077 ceph-mon[29247]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:38.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.057+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc40810e620 con 0x7fc4080fbad0 2024-02-19T18:51:38.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.058+0000 7fc3fdffb700 1 -- 172.21.15.104:0/1150390449 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fc3f8011370 con 0x7fc4080fbad0 2024-02-19T18:51:38.074 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:38.074 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.059+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc3f4036820 msgr2=0x7fc3f4038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.059+0000 7fc40f328700 1 --2- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc3f4036820 0x7fc3f4038cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc404009990 tx=0x7fc404006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.059+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 msgr2=0x7fc40810cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.059+0000 7fc40f328700 1 --2- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 0x7fc40810cec0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009680 tx=0x7fc3f8009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.060+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 shutdown_connections 2024-02-19T18:51:38.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.060+0000 7fc40f328700 1 --2- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fc3f4036820 0x7fc3f4038cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.060+0000 7fc40f328700 1 --2- 172.21.15.104:0/1150390449 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc4080fbad0 0x7fc40810cec0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.060+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 >> 172.21.15.104:0/1150390449 conn(0x7fc4080f7190 msgr2=0x7fc4080f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.060+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 shutdown_connections 2024-02-19T18:51:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:38.060+0000 7fc40f328700 1 -- 172.21.15.104:0/1150390449 wait complete. 2024-02-19T18:51:38.077 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:39.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:38 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1150390449' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:39.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:38 smithi077 ceph-mon[29247]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:39.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:39.519 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:39.764 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:39.812 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:41.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.058+0000 7fb242deb700 1 -- 172.21.15.104:0/2987056979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 msgr2=0x7fb23c0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:41.075 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.058+0000 7fb242deb700 1 --2- 172.21.15.104:0/2987056979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 0x7fb23c0fcfe0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb234009c60 tx=0x7fb234004d10 comp rx=0 tx=0).stop 2024-02-19T18:51:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.058+0000 7fb242deb700 1 -- 172.21.15.104:0/2987056979 shutdown_connections 2024-02-19T18:51:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.058+0000 7fb242deb700 1 --2- 172.21.15.104:0/2987056979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 0x7fb23c0fcfe0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.058+0000 7fb242deb700 1 -- 172.21.15.104:0/2987056979 >> 172.21.15.104:0/2987056979 conn(0x7fb23c0f8290 msgr2=0x7fb23c0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.059+0000 7fb242deb700 1 -- 172.21.15.104:0/2987056979 shutdown_connections 2024-02-19T18:51:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.059+0000 7fb242deb700 1 -- 172.21.15.104:0/2987056979 wait complete. 2024-02-19T18:51:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.059+0000 7fb242deb700 1 Processor -- start 2024-02-19T18:51:41.076 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.059+0000 7fb242deb700 1 -- start start 2024-02-19T18:51:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.059+0000 7fb242deb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 0x7fb23c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.059+0000 7fb242deb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23c10d400 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.059+0000 7fb240b87700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 0x7fb23c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb240b87700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 0x7fb23c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:45902/0 (socket says 172.21.15.104:45902) 2024-02-19T18:51:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb240b87700 1 -- 172.21.15.104:0/3694872444 learned_addr learned my addr 172.21.15.104:0/3694872444 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb240b87700 1 -- 172.21.15.104:0/3694872444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb234009940 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.077 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb240b87700 1 --2- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 0x7fb23c10cec0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb234015040 tx=0x7fb234004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb2397fa700 1 -- 172.21.15.104:0/3694872444 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb234007c80 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb2397fa700 1 -- 172.21.15.104:0/3694872444 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb23401d910 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23c10d600 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb2397fa700 1 -- 172.21.15.104:0/3694872444 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2340129f0 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.060+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23c10e2a0 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.061+0000 7fb2397fa700 1 -- 172.21.15.104:0/3694872444 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fb234019040 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.061+0000 7fb2397fa700 1 --2- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2280368c0 0x7fb228038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.061+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb23c05f5c0 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.061+0000 7fb2397fa700 1 -- 172.21.15.104:0/3694872444 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb23404a480 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.061+0000 7fb23bfff700 1 --2- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2280368c0 0x7fb228038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:41.079 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.062+0000 7fb23bfff700 1 --2- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2280368c0 0x7fb228038d70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009990 tx=0x7fb22c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:41.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.066+0000 7fb2397fa700 1 -- 172.21.15.104:0/3694872444 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb234026400 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.320+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb23c10edb0 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.336 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.321+0000 7fb2397fa700 1 -- 172.21.15.104:0/3694872444 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb234017030 con 0x7fb23c0fcbd0 2024-02-19T18:51:41.337 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:41.337 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.323+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2280368c0 msgr2=0x7fb228038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.323+0000 7fb242deb700 1 --2- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2280368c0 0x7fb228038d70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb22c009990 tx=0x7fb22c006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.323+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 msgr2=0x7fb23c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.323+0000 7fb242deb700 1 --2- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 0x7fb23c10cec0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb234015040 tx=0x7fb234004d10 comp rx=0 tx=0).stop 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.324+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 shutdown_connections 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.324+0000 7fb242deb700 1 --2- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2280368c0 0x7fb228038d70 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.324+0000 7fb242deb700 1 --2- 172.21.15.104:0/3694872444 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb23c0fcbd0 0x7fb23c10cec0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.324+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 >> 172.21.15.104:0/3694872444 conn(0x7fb23c0f8290 msgr2=0x7fb23c0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.324+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 shutdown_connections 2024-02-19T18:51:41.339 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:41.324+0000 7fb242deb700 1 -- 172.21.15.104:0/3694872444 wait complete. 2024-02-19T18:51:41.340 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:41.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:41 smithi077 ceph-mon[29247]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:42.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:42 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3694872444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:42.833 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:42.833 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:43.079 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:43.126 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:43.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:43 smithi077 ceph-mon[29247]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 -- 172.21.15.104:0/2091700476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 msgr2=0x7f95f80fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 --2- 172.21.15.104:0/2091700476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 0x7f95f80fbed0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f95e80048c0 tx=0x7f95e8009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 -- 172.21.15.104:0/2091700476 shutdown_connections 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 --2- 172.21.15.104:0/2091700476 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 0x7f95f80fbed0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 -- 172.21.15.104:0/2091700476 >> 172.21.15.104:0/2091700476 conn(0x7f95f80f71a0 msgr2=0x7f95f80f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 -- 172.21.15.104:0/2091700476 shutdown_connections 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 -- 172.21.15.104:0/2091700476 wait complete. 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 Processor -- start 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.487+0000 7f95fcd94700 1 -- start start 2024-02-19T18:51:44.504 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.488+0000 7f95fcd94700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 0x7f95f810fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.488+0000 7f95fcd94700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f8066810 con 0x7f95f80fbac0 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.488+0000 7f95f659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 0x7f95f810fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.488+0000 7f95f659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 0x7f95f810fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:45922/0 (socket says 172.21.15.104:45922) 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.488+0000 7f95f659c700 1 -- 172.21.15.104:0/1271898134 learned_addr learned my addr 172.21.15.104:0/1271898134 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.488+0000 7f95f659c700 1 -- 172.21.15.104:0/1271898134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e8005040 con 0x7f95f80fbac0 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.488+0000 7f95f659c700 1 --2- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 0x7f95f810fbd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009680 tx=0x7f95e8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.488+0000 7f95e6ffd700 1 -- 172.21.15.104:0/1271898134 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95e800bed0 con 0x7f95f80fbac0 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.489+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f8066a10 con 0x7f95f80fbac0 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.489+0000 7f95e6ffd700 1 -- 172.21.15.104:0/1271898134 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95e800b5c0 con 0x7f95f80fbac0 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.489+0000 7f95e6ffd700 1 -- 172.21.15.104:0/1271898134 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95e801c5b0 con 0x7f95f80fbac0 2024-02-19T18:51:44.505 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.489+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f8110350 con 0x7f95f80fbac0 2024-02-19T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.489+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f8105890 con 0x7f95f80fbac0 2024-02-19T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.489+0000 7f95e6ffd700 1 -- 172.21.15.104:0/1271898134 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f95e8013a80 con 0x7f95f80fbac0 2024-02-19T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.490+0000 7f95e6ffd700 1 --2- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f95e0036820 0x7f95e0038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:44.506 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.490+0000 7f95e6ffd700 1 -- 172.21.15.104:0/1271898134 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f95e8049ab0 con 0x7f95f80fbac0 2024-02-19T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.490+0000 7f95f5d9b700 1 --2- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f95e0036820 0x7f95e0038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:44.507 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.490+0000 7f95f5d9b700 1 --2- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f95e0036820 0x7f95e0038cd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009990 tx=0x7f95ec006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:44.510 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.495+0000 7f95e6ffd700 1 -- 172.21.15.104:0/1271898134 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f95e8011940 con 0x7f95f80fbac0 2024-02-19T18:51:44.766 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.751+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95f804bed0 con 0x7f95f80fbac0 2024-02-19T18:51:44.767 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.752+0000 7f95e6ffd700 1 -- 172.21.15.104:0/1271898134 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f95e8011370 con 0x7f95f80fbac0 2024-02-19T18:51:44.767 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:44.767 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:44.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f95e0036820 msgr2=0x7f95e0038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:44.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 --2- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f95e0036820 0x7f95e0038cd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009990 tx=0x7f95ec006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:44.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 msgr2=0x7f95f810fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:44.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 --2- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 0x7f95f810fbd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009680 tx=0x7f95e8009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:44.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 shutdown_connections 2024-02-19T18:51:44.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 --2- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f95e0036820 0x7f95e0038cd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:44.769 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 --2- 172.21.15.104:0/1271898134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f95f80fbac0 0x7f95f810fbd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:44.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 >> 172.21.15.104:0/1271898134 conn(0x7f95f80f71a0 msgr2=0x7f95f80f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:44.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 shutdown_connections 2024-02-19T18:51:44.770 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:44.754+0000 7f95fcd94700 1 -- 172.21.15.104:0/1271898134 wait complete. 2024-02-19T18:51:44.770 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:45.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:45 smithi077 ceph-mon[29247]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:45.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:45 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1271898134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:46.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:46.342 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:46.597 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:46.643 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:47.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:47 smithi077 ceph-mon[29247]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:47.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.958+0000 7f2ec622f700 1 -- 172.21.15.104:0/3035261419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 msgr2=0x7f2ec00fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:47.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.958+0000 7f2ec622f700 1 --2- 172.21.15.104:0/3035261419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 0x7f2ec00fbee0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00048c0 tx=0x7f2eb0009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:47.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.958+0000 7f2ec622f700 1 -- 172.21.15.104:0/3035261419 shutdown_connections 2024-02-19T18:51:47.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.958+0000 7f2ec622f700 1 --2- 172.21.15.104:0/3035261419 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 0x7f2ec00fbee0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:47.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.958+0000 7f2ec622f700 1 -- 172.21.15.104:0/3035261419 >> 172.21.15.104:0/3035261419 conn(0x7f2ec00f7190 msgr2=0x7f2ec00f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:47.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.958+0000 7f2ec622f700 1 -- 172.21.15.104:0/3035261419 shutdown_connections 2024-02-19T18:51:47.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.959+0000 7f2ec622f700 1 -- 172.21.15.104:0/3035261419 wait complete. 2024-02-19T18:51:47.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.959+0000 7f2ec622f700 1 Processor -- start 2024-02-19T18:51:47.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.959+0000 7f2ec622f700 1 -- start start 2024-02-19T18:51:47.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.959+0000 7f2ec622f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 0x7f2ec010fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:47.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.959+0000 7f2ec622f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec0065d80 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.960+0000 7f2ebf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 0x7f2ec010fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.960+0000 7f2ebf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 0x7f2ec010fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:36344/0 (socket says 172.21.15.104:36344) 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.960+0000 7f2ebf7fe700 1 -- 172.21.15.104:0/2540114062 learned_addr learned my addr 172.21.15.104:0/2540114062 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.960+0000 7f2ebf7fe700 1 -- 172.21.15.104:0/2540114062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb0005040 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.960+0000 7f2ebf7fe700 1 --2- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 0x7f2ec010fbb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009680 tx=0x7f2eb0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.960+0000 7f2ebcff9700 1 -- 172.21.15.104:0/2540114062 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb000bed0 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.961+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec0065f80 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.961+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec0110330 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.961+0000 7f2ebcff9700 1 -- 172.21.15.104:0/2540114062 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2eb000b5c0 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.961+0000 7f2ebcff9700 1 -- 172.21.15.104:0/2540114062 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2eb00255c0 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.961+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec01058a0 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.962+0000 7f2ebcff9700 1 -- 172.21.15.104:0/2540114062 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f2eb0013420 con 0x7f2ec00fbad0 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.962+0000 7f2ebcff9700 1 --2- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2eac036870 0x7f2eac038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:47.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.962+0000 7f2ebcff9700 1 -- 172.21.15.104:0/2540114062 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2eb0049af0 con 0x7f2ec00fbad0 2024-02-19T18:51:47.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.962+0000 7f2eb6ffd700 1 --2- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2eac036870 0x7f2eac038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:47.979 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.962+0000 7f2eb6ffd700 1 --2- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2eac036870 0x7f2eac038d20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009990 tx=0x7f2ea8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:47.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:47.967+0000 7f2ebcff9700 1 -- 172.21.15.104:0/2540114062 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2eb00139f0 con 0x7f2ec00fbad0 2024-02-19T18:51:48.240 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.225+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ec004bed0 con 0x7f2ec00fbad0 2024-02-19T18:51:48.241 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.226+0000 7f2ebcff9700 1 -- 172.21.15.104:0/2540114062 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2eb0023300 con 0x7f2ec00fbad0 2024-02-19T18:51:48.241 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:48.242 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:48.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2eac036870 msgr2=0x7f2eac038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:48.243 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 --2- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2eac036870 0x7f2eac038d20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ea8009990 tx=0x7f2ea8006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 msgr2=0x7f2ec010fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 --2- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 0x7f2ec010fbb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009680 tx=0x7f2eb0009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 shutdown_connections 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 --2- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f2eac036870 0x7f2eac038d20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 --2- 172.21.15.104:0/2540114062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2ec00fbad0 0x7f2ec010fbb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 >> 172.21.15.104:0/2540114062 conn(0x7f2ec00f7190 msgr2=0x7f2ec00f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 shutdown_connections 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:48.228+0000 7f2ec622f700 1 -- 172.21.15.104:0/2540114062 wait complete. 2024-02-19T18:51:48.244 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:49.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:48 smithi077 ceph-mon[29247]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:49.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:48 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2540114062' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:49.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:49.813 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:50.063 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:50.110 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.476+0000 7f17202b2700 1 -- 172.21.15.104:0/343465775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 msgr2=0x7f17180fb290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.476+0000 7f17202b2700 1 --2- 172.21.15.104:0/343465775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 0x7f17180fb290 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f17140048c0 tx=0x7f1714009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.476+0000 7f17202b2700 1 -- 172.21.15.104:0/343465775 shutdown_connections 2024-02-19T18:51:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.476+0000 7f17202b2700 1 --2- 172.21.15.104:0/343465775 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 0x7f17180fb290 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.476+0000 7f17202b2700 1 -- 172.21.15.104:0/343465775 >> 172.21.15.104:0/343465775 conn(0x7f17180f6920 msgr2=0x7f17180f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.476+0000 7f17202b2700 1 -- 172.21.15.104:0/343465775 shutdown_connections 2024-02-19T18:51:51.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.476+0000 7f17202b2700 1 -- 172.21.15.104:0/343465775 wait complete. 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.477+0000 7f17202b2700 1 Processor -- start 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.477+0000 7f17202b2700 1 -- start start 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.477+0000 7f17202b2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 0x7f17180fd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.477+0000 7f17202b2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17180fbe20 con 0x7f17180fae80 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.477+0000 7f171e04e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 0x7f17180fd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.477+0000 7f171e04e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 0x7f17180fd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:36362/0 (socket says 172.21.15.104:36362) 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.477+0000 7f171e04e700 1 -- 172.21.15.104:0/1629881262 learned_addr learned my addr 172.21.15.104:0/1629881262 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.477+0000 7f171e04e700 1 -- 172.21.15.104:0/1629881262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714005040 con 0x7f17180fae80 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.478+0000 7f171e04e700 1 --2- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 0x7f17180fd9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1714009040 tx=0x7f1714009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.478+0000 7f170effd700 1 -- 172.21.15.104:0/1629881262 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171400bed0 con 0x7f17180fae80 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.478+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17180fc020 con 0x7f17180fae80 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.478+0000 7f170effd700 1 -- 172.21.15.104:0/1629881262 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f171400b5c0 con 0x7f17180fae80 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.478+0000 7f170effd700 1 -- 172.21.15.104:0/1629881262 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f171401c5b0 con 0x7f17180fae80 2024-02-19T18:51:51.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.478+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17180fc510 con 0x7f17180fae80 2024-02-19T18:51:51.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.479+0000 7f170effd700 1 -- 172.21.15.104:0/1629881262 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f1714004d30 con 0x7f17180fae80 2024-02-19T18:51:51.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.479+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1718106930 con 0x7f17180fae80 2024-02-19T18:51:51.495 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.479+0000 7f170effd700 1 --2- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f1704036870 0x7f1704038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.479+0000 7f170effd700 1 -- 172.21.15.104:0/1629881262 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1714049b40 con 0x7f17180fae80 2024-02-19T18:51:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.479+0000 7f171d84d700 1 --2- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f1704036870 0x7f1704038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:51.496 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.479+0000 7f171d84d700 1 --2- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f1704036870 0x7f1704038d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1708009990 tx=0x7f1708006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:51.499 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.484+0000 7f170effd700 1 -- 172.21.15.104:0/1629881262 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1714011550 con 0x7f17180fae80 2024-02-19T18:51:51.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:51 smithi077 ceph-mon[29247]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:51.752 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.737+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f171804bed0 con 0x7f17180fae80 2024-02-19T18:51:51.753 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.738+0000 7f170effd700 1 -- 172.21.15.104:0/1629881262 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1714011370 con 0x7f17180fae80 2024-02-19T18:51:51.753 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:51.753 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:51.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f1704036870 msgr2=0x7f1704038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:51.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 --2- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f1704036870 0x7f1704038d20 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1708009990 tx=0x7f1708006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:51.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 msgr2=0x7f17180fd9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:51.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 --2- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 0x7f17180fd9a0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1714009040 tx=0x7f1714009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:51.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 shutdown_connections 2024-02-19T18:51:51.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 --2- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f1704036870 0x7f1704038d20 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:51.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 --2- 172.21.15.104:0/1629881262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f17180fae80 0x7f17180fd9a0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:51.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 >> 172.21.15.104:0/1629881262 conn(0x7f17180f6920 msgr2=0x7f17180f76e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:51.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 shutdown_connections 2024-02-19T18:51:51.756 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:51.740+0000 7f17202b2700 1 -- 172.21.15.104:0/1629881262 wait complete. 2024-02-19T18:51:51.756 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:52.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:52 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1629881262' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:53.248 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:53.248 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:53.498 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:53.544 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:53.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:53 smithi077 ceph-mon[29247]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:55.021 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.005+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3075773502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 msgr2=0x7f8c680fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.005+0000 7f8c6fb2c700 1 --2- 172.21.15.104:0/3075773502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 0x7f8c680fcfd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8c64009c60 tx=0x7f8c64004d10 comp rx=0 tx=0).stop 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.005+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3075773502 shutdown_connections 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.005+0000 7f8c6fb2c700 1 --2- 172.21.15.104:0/3075773502 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 0x7f8c680fcfd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.005+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3075773502 >> 172.21.15.104:0/3075773502 conn(0x7f8c680f82a0 msgr2=0x7f8c680fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.005+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3075773502 shutdown_connections 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.005+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3075773502 wait complete. 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.006+0000 7f8c6fb2c700 1 Processor -- start 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.006+0000 7f8c6fb2c700 1 -- start start 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.006+0000 7f8c6fb2c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 0x7f8c6810ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.006+0000 7f8c6fb2c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6810d3d0 con 0x7f8c680fcbc0 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.006+0000 7f8c6d8c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 0x7f8c6810ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.006+0000 7f8c6d8c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 0x7f8c6810ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:36390/0 (socket says 172.21.15.104:36390) 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.006+0000 7f8c6d8c8700 1 -- 172.21.15.104:0/3674000314 learned_addr learned my addr 172.21.15.104:0/3674000314 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:55.022 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.006+0000 7f8c6d8c8700 1 -- 172.21.15.104:0/3674000314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c64009940 con 0x7f8c680fcbc0 2024-02-19T18:51:55.023 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.007+0000 7f8c6d8c8700 1 --2- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 0x7f8c6810ce90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8c64015040 tx=0x7f8c64004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.007+0000 7f8c5e7fc700 1 -- 172.21.15.104:0/3674000314 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c64007c80 con 0x7f8c680fcbc0 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.007+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6810d5d0 con 0x7f8c680fcbc0 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.007+0000 7f8c5e7fc700 1 -- 172.21.15.104:0/3674000314 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8c6401d910 con 0x7f8c680fcbc0 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.007+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6810e270 con 0x7f8c680fcbc0 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.007+0000 7f8c5e7fc700 1 -- 172.21.15.104:0/3674000314 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8c640129f0 con 0x7f8c680fcbc0 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.008+0000 7f8c5e7fc700 1 -- 172.21.15.104:0/3674000314 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f8c64019040 con 0x7f8c680fcbc0 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.008+0000 7f8c5e7fc700 1 --2- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8c54036870 0x7f8c54038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.008+0000 7f8c5e7fc700 1 -- 172.21.15.104:0/3674000314 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8c6404a380 con 0x7f8c680fcbc0 2024-02-19T18:51:55.024 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.008+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6805f5c0 con 0x7f8c680fcbc0 2024-02-19T18:51:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.008+0000 7f8c6d0c7700 1 --2- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8c54036870 0x7f8c54038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:55.025 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.009+0000 7f8c6d0c7700 1 --2- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8c54036870 0x7f8c54038d20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009990 tx=0x7f8c58006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:55.028 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.013+0000 7f8c5e7fc700 1 -- 172.21.15.104:0/3674000314 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8c64017020 con 0x7f8c680fcbc0 2024-02-19T18:51:55.278 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.263+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8c6804bed0 con 0x7f8c680fcbc0 2024-02-19T18:51:55.279 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.264+0000 7f8c5e7fc700 1 -- 172.21.15.104:0/3674000314 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8c64035090 con 0x7f8c680fcbc0 2024-02-19T18:51:55.279 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:55.279 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8c54036870 msgr2=0x7f8c54038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 --2- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8c54036870 0x7f8c54038d20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8c58009990 tx=0x7f8c58006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 msgr2=0x7f8c6810ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 --2- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 0x7f8c6810ce90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8c64015040 tx=0x7f8c64004d10 comp rx=0 tx=0).stop 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 shutdown_connections 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 --2- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8c54036870 0x7f8c54038d20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 --2- 172.21.15.104:0/3674000314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8c680fcbc0 0x7f8c6810ce90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 >> 172.21.15.104:0/3674000314 conn(0x7f8c680f82a0 msgr2=0x7f8c680f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 shutdown_connections 2024-02-19T18:51:55.282 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:55.266+0000 7f8c6fb2c700 1 -- 172.21.15.104:0/3674000314 wait complete. 2024-02-19T18:51:55.283 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:55.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:55 smithi077 ceph-mon[29247]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:56.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:56 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3674000314' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:51:56.885 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:51:56.886 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:51:57.126 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:57.175 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:51:57.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:57 smithi077 ceph-mon[29247]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:58.401 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.384+0000 7f7024674700 1 -- 172.21.15.104:0/3015024092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 msgr2=0x7f701c0fd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.384+0000 7f7024674700 1 --2- 172.21.15.104:0/3015024092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 0x7f701c0fd4c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f70180048c0 tx=0x7f7018009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.384+0000 7f7024674700 1 -- 172.21.15.104:0/3015024092 shutdown_connections 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.384+0000 7f7024674700 1 --2- 172.21.15.104:0/3015024092 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 0x7f701c0fd4c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.384+0000 7f7024674700 1 -- 172.21.15.104:0/3015024092 >> 172.21.15.104:0/3015024092 conn(0x7f701c0f6920 msgr2=0x7f701c0f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.384+0000 7f7024674700 1 -- 172.21.15.104:0/3015024092 shutdown_connections 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.384+0000 7f7024674700 1 -- 172.21.15.104:0/3015024092 wait complete. 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.385+0000 7f7024674700 1 Processor -- start 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.385+0000 7f7024674700 1 -- start start 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.385+0000 7f7024674700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 0x7f701c10cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.385+0000 7f7024674700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c10d480 con 0x7f701c0fd0b0 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.385+0000 7f7022410700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 0x7f701c10cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:58.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7022410700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 0x7f701c10cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:57924/0 (socket says 172.21.15.104:57924) 2024-02-19T18:51:58.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7022410700 1 -- 172.21.15.104:0/4188101776 learned_addr learned my addr 172.21.15.104:0/4188101776 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:51:58.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7022410700 1 -- 172.21.15.104:0/4188101776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7018005040 con 0x7f701c0fd0b0 2024-02-19T18:51:58.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7022410700 1 --2- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 0x7f701c10cf40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7018009f80 tx=0x7f7018009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:58.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7012ffd700 1 -- 172.21.15.104:0/4188101776 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7018004e60 con 0x7f701c0fd0b0 2024-02-19T18:51:58.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c10d680 con 0x7f701c0fd0b0 2024-02-19T18:51:58.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7012ffd700 1 -- 172.21.15.104:0/4188101776 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7018009650 con 0x7f701c0fd0b0 2024-02-19T18:51:58.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7012ffd700 1 -- 172.21.15.104:0/4188101776 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f701801c5b0 con 0x7f701c0fd0b0 2024-02-19T18:51:58.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.386+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c10e320 con 0x7f701c0fd0b0 2024-02-19T18:51:58.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.387+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c05f5c0 con 0x7f701c0fd0b0 2024-02-19T18:51:58.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.387+0000 7f7012ffd700 1 -- 172.21.15.104:0/4188101776 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f7018013980 con 0x7f701c0fd0b0 2024-02-19T18:51:58.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.387+0000 7f7012ffd700 1 --2- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7008036870 0x7f7008038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:51:58.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.387+0000 7f7012ffd700 1 -- 172.21.15.104:0/4188101776 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7018049520 con 0x7f701c0fd0b0 2024-02-19T18:51:58.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.388+0000 7f7021c0f700 1 --2- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7008036870 0x7f7008038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:51:58.404 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.388+0000 7f7021c0f700 1 --2- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7008036870 0x7f7008038d20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f700c009990 tx=0x7f700c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:51:58.408 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.393+0000 7f7012ffd700 1 -- 172.21.15.104:0/4188101776 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f70180137a0 con 0x7f701c0fd0b0 2024-02-19T18:51:58.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.646+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f701c10ee90 con 0x7f701c0fd0b0 2024-02-19T18:51:58.662 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.647+0000 7f7012ffd700 1 -- 172.21.15.104:0/4188101776 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7018021070 con 0x7f701c0fd0b0 2024-02-19T18:51:58.663 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:51:58.663 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:51:58.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7008036870 msgr2=0x7f7008038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:58.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 --2- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7008036870 0x7f7008038d20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f700c009990 tx=0x7f700c006e40 comp rx=0 tx=0).stop 2024-02-19T18:51:58.664 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 msgr2=0x7f701c10cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:51:58.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 --2- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 0x7f701c10cf40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7018009f80 tx=0x7f7018009e30 comp rx=0 tx=0).stop 2024-02-19T18:51:58.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 shutdown_connections 2024-02-19T18:51:58.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 --2- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7008036870 0x7f7008038d20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:58.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 --2- 172.21.15.104:0/4188101776 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f701c0fd0b0 0x7f701c10cf40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:51:58.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 >> 172.21.15.104:0/4188101776 conn(0x7f701c0f6920 msgr2=0x7f701c0fb4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:51:58.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 shutdown_connections 2024-02-19T18:51:58.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:51:58.649+0000 7f7024674700 1 -- 172.21.15.104:0/4188101776 wait complete. 2024-02-19T18:51:58.665 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:51:59.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:58 smithi077 ceph-mon[29247]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:51:59.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:51:58 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/4188101776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:00.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:00.215 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:00.466 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:00.514 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:01.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:01 smithi077 ceph-mon[29247]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.936+0000 7f7ed259e700 1 -- 172.21.15.104:0/977266522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 msgr2=0x7f7ecc0fe470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.936+0000 7f7ed259e700 1 --2- 172.21.15.104:0/977266522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 0x7f7ecc0fe470 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7ec80048c0 tx=0x7f7ec8009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.936+0000 7f7ed259e700 1 -- 172.21.15.104:0/977266522 shutdown_connections 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.936+0000 7f7ed259e700 1 --2- 172.21.15.104:0/977266522 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 0x7f7ecc0fe470 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.936+0000 7f7ed259e700 1 -- 172.21.15.104:0/977266522 >> 172.21.15.104:0/977266522 conn(0x7f7ecc0f6740 msgr2=0x7f7ecc0f8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed259e700 1 -- 172.21.15.104:0/977266522 shutdown_connections 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed259e700 1 -- 172.21.15.104:0/977266522 wait complete. 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed259e700 1 Processor -- start 2024-02-19T18:52:01.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed259e700 1 -- start start 2024-02-19T18:52:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed259e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 0x7f7ecc0fd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed259e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc0fbd20 con 0x7f7ecc0fe060 2024-02-19T18:52:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed159c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 0x7f7ecc0fd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed159c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 0x7f7ecc0fd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:57934/0 (socket says 172.21.15.104:57934) 2024-02-19T18:52:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.937+0000 7f7ed159c700 1 -- 172.21.15.104:0/1439192664 learned_addr learned my addr 172.21.15.104:0/1439192664 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.938+0000 7f7ed159c700 1 -- 172.21.15.104:0/1439192664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec8005040 con 0x7f7ecc0fe060 2024-02-19T18:52:01.954 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.938+0000 7f7ed159c700 1 --2- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 0x7f7ecc0fd8a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009b80 tx=0x7f7ec8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:01.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.938+0000 7f7ec1ffb700 1 -- 172.21.15.104:0/1439192664 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ec800b5c0 con 0x7f7ecc0fe060 2024-02-19T18:52:01.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.938+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ecc0fbf20 con 0x7f7ecc0fe060 2024-02-19T18:52:01.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.938+0000 7f7ec1ffb700 1 -- 172.21.15.104:0/1439192664 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ec800be70 con 0x7f7ecc0fe060 2024-02-19T18:52:01.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.938+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ecc0fc390 con 0x7f7ecc0fe060 2024-02-19T18:52:01.955 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.938+0000 7f7ec1ffb700 1 -- 172.21.15.104:0/1439192664 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7ec801c600 con 0x7f7ecc0fe060 2024-02-19T18:52:01.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.939+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ecc106700 con 0x7f7ecc0fe060 2024-02-19T18:52:01.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.939+0000 7f7ec1ffb700 1 -- 172.21.15.104:0/1439192664 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f7ec8004d30 con 0x7f7ecc0fe060 2024-02-19T18:52:01.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.939+0000 7f7ec1ffb700 1 --2- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7eb8036870 0x7f7eb8038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:01.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.939+0000 7f7ec1ffb700 1 -- 172.21.15.104:0/1439192664 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7ec80499d0 con 0x7f7ecc0fe060 2024-02-19T18:52:01.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.940+0000 7f7ed0d9b700 1 --2- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7eb8036870 0x7f7eb8038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:01.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.940+0000 7f7ed0d9b700 1 --2- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7eb8036870 0x7f7eb8038d20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009990 tx=0x7f7ebc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:01.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:01.944+0000 7f7ec1ffb700 1 -- 172.21.15.104:0/1439192664 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7ec8025400 con 0x7f7ecc0fe060 2024-02-19T18:52:02.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.200+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ecc05f5c0 con 0x7f7ecc0fe060 2024-02-19T18:52:02.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.201+0000 7f7ec1ffb700 1 -- 172.21.15.104:0/1439192664 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7ec8011370 con 0x7f7ecc0fe060 2024-02-19T18:52:02.216 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:02.217 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:02.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7eb8036870 msgr2=0x7f7eb8038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:02.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 --2- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7eb8036870 0x7f7eb8038d20 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc009990 tx=0x7f7ebc006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:02.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 msgr2=0x7f7ecc0fd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:02.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 --2- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 0x7f7ecc0fd8a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8009b80 tx=0x7f7ec8009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:02.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 shutdown_connections 2024-02-19T18:52:02.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 --2- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f7eb8036870 0x7f7eb8038d20 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:02.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 --2- 172.21.15.104:0/1439192664 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ecc0fe060 0x7f7ecc0fd8a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:02.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 >> 172.21.15.104:0/1439192664 conn(0x7f7ecc0f6740 msgr2=0x7f7ecc0f75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:02.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 shutdown_connections 2024-02-19T18:52:02.219 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:02.203+0000 7f7ed259e700 1 -- 172.21.15.104:0/1439192664 wait complete. 2024-02-19T18:52:02.219 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:02.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:02 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1439192664' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:03.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:03 smithi077 ceph-mon[29247]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:03.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:03.785 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:04.035 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:04.084 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 -- 172.21.15.104:0/3406078015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 msgr2=0x7ff9a40fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 --2- 172.21.15.104:0/3406078015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 0x7ff9a40fcfe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00048c0 tx=0x7ff9a0009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 -- 172.21.15.104:0/3406078015 shutdown_connections 2024-02-19T18:52:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 --2- 172.21.15.104:0/3406078015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 0x7ff9a40fcfe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 -- 172.21.15.104:0/3406078015 >> 172.21.15.104:0/3406078015 conn(0x7ff9a40f8290 msgr2=0x7ff9a40fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 -- 172.21.15.104:0/3406078015 shutdown_connections 2024-02-19T18:52:05.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 -- 172.21.15.104:0/3406078015 wait complete. 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 Processor -- start 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 -- start start 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 0x7ff9a410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.400+0000 7ff9abb30700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a410d400 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff9a98cc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 0x7ff9a410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff9a98cc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 0x7ff9a410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:57950/0 (socket says 172.21.15.104:57950) 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff9a98cc700 1 -- 172.21.15.104:0/3439056473 learned_addr learned my addr 172.21.15.104:0/3439056473 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff9a98cc700 1 -- 172.21.15.104:0/3439056473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a0005040 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.417 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff9a98cc700 1 --2- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 0x7ff9a410cec0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009680 tx=0x7ff9a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff99a7fc700 1 -- 172.21.15.104:0/3439056473 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9a000bed0 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a410d600 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.418 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff99a7fc700 1 -- 172.21.15.104:0/3439056473 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9a000b5c0 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a410e2a0 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.401+0000 7ff99a7fc700 1 -- 172.21.15.104:0/3439056473 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9a001c5b0 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.402+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a405f5c0 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.402+0000 7ff99a7fc700 1 -- 172.21.15.104:0/3439056473 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ff9a0013a80 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.402+0000 7ff99a7fc700 1 --2- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff990036820 0x7ff990038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.402+0000 7ff99a7fc700 1 -- 172.21.15.104:0/3439056473 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff9a0049b70 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.403+0000 7ff9a90cb700 1 --2- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff990036820 0x7ff990038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:05.419 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.403+0000 7ff9a90cb700 1 --2- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff990036820 0x7ff990038cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff994009990 tx=0x7ff994006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:05.422 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.407+0000 7ff99a7fc700 1 -- 172.21.15.104:0/3439056473 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff9a0011940 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:05 smithi077 ceph-mon[29247]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:05.679 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.664+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff9a410ee10 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.680 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.665+0000 7ff99a7fc700 1 -- 172.21.15.104:0/3439056473 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff9a0011370 con 0x7ff9a40fcbd0 2024-02-19T18:52:05.680 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:05.680 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff990036820 msgr2=0x7ff990038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 --2- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff990036820 0x7ff990038cd0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ff994009990 tx=0x7ff994006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 msgr2=0x7ff9a410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 --2- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 0x7ff9a410cec0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0009680 tx=0x7ff9a0009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 shutdown_connections 2024-02-19T18:52:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 --2- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff990036820 0x7ff990038cd0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 --2- 172.21.15.104:0/3439056473 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff9a40fcbd0 0x7ff9a410cec0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:05.682 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 >> 172.21.15.104:0/3439056473 conn(0x7ff9a40f8290 msgr2=0x7ff9a40f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 shutdown_connections 2024-02-19T18:52:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:05.667+0000 7ff9abb30700 1 -- 172.21.15.104:0/3439056473 wait complete. 2024-02-19T18:52:05.683 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:06.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:06 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3439056473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:07.246 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:07.246 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:07.493 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:07.540 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:07.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:07 smithi077 ceph-mon[29247]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 -- 172.21.15.104:0/1853588341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 msgr2=0x7fdac00ff790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 --2- 172.21.15.104:0/1853588341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 0x7fdac00ff790 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdab40048c0 tx=0x7fdab4009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 -- 172.21.15.104:0/1853588341 shutdown_connections 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 --2- 172.21.15.104:0/1853588341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 0x7fdac00ff790 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 -- 172.21.15.104:0/1853588341 >> 172.21.15.104:0/1853588341 conn(0x7fdac00f71a0 msgr2=0x7fdac00f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 -- 172.21.15.104:0/1853588341 shutdown_connections 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 -- 172.21.15.104:0/1853588341 wait complete. 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 Processor -- start 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.843+0000 7fdac6bf2700 1 -- start start 2024-02-19T18:52:08.861 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.844+0000 7fdac6bf2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 0x7fdac010cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:08.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.844+0000 7fdac6bf2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdac010d3e0 con 0x7fdac0066860 2024-02-19T18:52:08.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.844+0000 7fdac498e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 0x7fdac010cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:08.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.844+0000 7fdac498e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 0x7fdac010cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:41600/0 (socket says 172.21.15.104:41600) 2024-02-19T18:52:08.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.844+0000 7fdac498e700 1 -- 172.21.15.104:0/2805909818 learned_addr learned my addr 172.21.15.104:0/2805909818 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:08.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.844+0000 7fdac498e700 1 -- 172.21.15.104:0/2805909818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab4005040 con 0x7fdac0066860 2024-02-19T18:52:08.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.844+0000 7fdac498e700 1 --2- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 0x7fdac010cea0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009680 tx=0x7fdab4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:08.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.844+0000 7fdabd7fa700 1 -- 172.21.15.104:0/2805909818 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdab400b5c0 con 0x7fdac0066860 2024-02-19T18:52:08.862 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.845+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdac010d5e0 con 0x7fdac0066860 2024-02-19T18:52:08.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.845+0000 7fdabd7fa700 1 -- 172.21.15.104:0/2805909818 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdab4004e60 con 0x7fdac0066860 2024-02-19T18:52:08.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.845+0000 7fdabd7fa700 1 -- 172.21.15.104:0/2805909818 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdab401c590 con 0x7fdac0066860 2024-02-19T18:52:08.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.845+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdac010e280 con 0x7fdac0066860 2024-02-19T18:52:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.845+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdac005f5c0 con 0x7fdac0066860 2024-02-19T18:52:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.845+0000 7fdabd7fa700 1 -- 172.21.15.104:0/2805909818 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fdab4013420 con 0x7fdac0066860 2024-02-19T18:52:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.846+0000 7fdabd7fa700 1 --2- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fdab0036870 0x7fdab0038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.846+0000 7fdabd7fa700 1 -- 172.21.15.104:0/2805909818 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdab4049bb0 con 0x7fdac0066860 2024-02-19T18:52:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.846+0000 7fdabffff700 1 --2- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fdab0036870 0x7fdab0038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:08.865 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.846+0000 7fdabffff700 1 --2- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fdab0036870 0x7fdab0038d20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009990 tx=0x7fdaac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:08.867 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:08.851+0000 7fdabd7fa700 1 -- 172.21.15.104:0/2805909818 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fdab4025400 con 0x7fdac0066860 2024-02-19T18:52:09.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.105+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdac010edd0 con 0x7fdac0066860 2024-02-19T18:52:09.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.106+0000 7fdabd7fa700 1 -- 172.21.15.104:0/2805909818 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fdab4011370 con 0x7fdac0066860 2024-02-19T18:52:09.122 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:09.122 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fdab0036870 msgr2=0x7fdab0038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 --2- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fdab0036870 0x7fdab0038d20 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdaac009990 tx=0x7fdaac006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 msgr2=0x7fdac010cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 --2- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 0x7fdac010cea0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009680 tx=0x7fdab4009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 shutdown_connections 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 --2- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fdab0036870 0x7fdab0038d20 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 --2- 172.21.15.104:0/2805909818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdac0066860 0x7fdac010cea0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 >> 172.21.15.104:0/2805909818 conn(0x7fdac00f71a0 msgr2=0x7fdac00f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 shutdown_connections 2024-02-19T18:52:09.125 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:09.108+0000 7fdac6bf2700 1 -- 172.21.15.104:0/2805909818 wait complete. 2024-02-19T18:52:09.126 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:09.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:08 smithi077 ceph-mon[29247]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:10.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:09 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2805909818' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:10.744 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:10.744 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:10.992 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:11.038 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:11.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:10 smithi077 ceph-mon[29247]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 -- 172.21.15.104:0/1729415931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 msgr2=0x7fa8d00fb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 --2- 172.21.15.104:0/1729415931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 0x7fa8d00fb2b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc0048c0 tx=0x7fa8cc009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 -- 172.21.15.104:0/1729415931 shutdown_connections 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 --2- 172.21.15.104:0/1729415931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 0x7fa8d00fb2b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 -- 172.21.15.104:0/1729415931 >> 172.21.15.104:0/1729415931 conn(0x7fa8d00f6920 msgr2=0x7fa8d00f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 -- 172.21.15.104:0/1729415931 shutdown_connections 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 -- 172.21.15.104:0/1729415931 wait complete. 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 Processor -- start 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 -- start start 2024-02-19T18:52:12.491 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 0x7fa8d010cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:12.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.474+0000 7fa8d7754700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d010d450 con 0x7fa8d00faea0 2024-02-19T18:52:12.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8d54f0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 0x7fa8d010cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:12.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8d54f0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 0x7fa8d010cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:41612/0 (socket says 172.21.15.104:41612) 2024-02-19T18:52:12.492 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8d54f0700 1 -- 172.21.15.104:0/898503980 learned_addr learned my addr 172.21.15.104:0/898503980 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8d54f0700 1 -- 172.21.15.104:0/898503980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8cc005040 con 0x7fa8d00faea0 2024-02-19T18:52:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8d54f0700 1 --2- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 0x7fa8d010cf10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009330 tx=0x7fa8cc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8c5ffb700 1 -- 172.21.15.104:0/898503980 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8cc004e60 con 0x7fa8d00faea0 2024-02-19T18:52:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d010d650 con 0x7fa8d00faea0 2024-02-19T18:52:12.493 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8c5ffb700 1 -- 172.21.15.104:0/898503980 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8cc0096b0 con 0x7fa8d00faea0 2024-02-19T18:52:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.475+0000 7fa8c5ffb700 1 -- 172.21.15.104:0/898503980 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8cc01c5b0 con 0x7fa8d00faea0 2024-02-19T18:52:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.476+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d010e2f0 con 0x7fa8d00faea0 2024-02-19T18:52:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.476+0000 7fa8c5ffb700 1 -- 172.21.15.104:0/898503980 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fa8cc01c820 con 0x7fa8d00faea0 2024-02-19T18:52:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.477+0000 7fa8c5ffb700 1 --2- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa8bc036870 0x7fa8bc038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.477+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d005f5c0 con 0x7fa8d00faea0 2024-02-19T18:52:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.477+0000 7fa8c5ffb700 1 -- 172.21.15.104:0/898503980 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa8cc049ea0 con 0x7fa8d00faea0 2024-02-19T18:52:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.477+0000 7fa8d4cef700 1 --2- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa8bc036870 0x7fa8bc038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:12.494 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.477+0000 7fa8d4cef700 1 --2- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa8bc036870 0x7fa8bc038d20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009990 tx=0x7fa8c0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:12.498 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.482+0000 7fa8c5ffb700 1 -- 172.21.15.104:0/898503980 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa8cc04e070 con 0x7fa8d00faea0 2024-02-19T18:52:12.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.734+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8d010ee40 con 0x7fa8d00faea0 2024-02-19T18:52:12.751 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.735+0000 7fa8c5ffb700 1 -- 172.21.15.104:0/898503980 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa8cc011ba0 con 0x7fa8d00faea0 2024-02-19T18:52:12.751 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:12.752 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa8bc036870 msgr2=0x7fa8bc038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 --2- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa8bc036870 0x7fa8bc038d20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009990 tx=0x7fa8c0006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 msgr2=0x7fa8d010cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 --2- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 0x7fa8d010cf10 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa8cc009330 tx=0x7fa8cc009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 shutdown_connections 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 --2- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa8bc036870 0x7fa8bc038d20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 --2- 172.21.15.104:0/898503980 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8d00faea0 0x7fa8d010cf10 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 >> 172.21.15.104:0/898503980 conn(0x7fa8d00f6920 msgr2=0x7fa8d00f76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 shutdown_connections 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:12.738+0000 7fa8d7754700 1 -- 172.21.15.104:0/898503980 wait complete. 2024-02-19T18:52:12.755 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:13.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:13 smithi077 ceph-mon[29247]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:13.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:13 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/898503980' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:14.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:14.348 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:14.599 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:14.646 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:15.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:15 smithi077 ceph-mon[29247]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:15.958 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 -- 172.21.15.104:0/2643895910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 msgr2=0x7f5aac0fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 --2- 172.21.15.104:0/2643895910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 0x7f5aac0fcfe0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009c60 tx=0x7f5aa4004d10 comp rx=0 tx=0).stop 2024-02-19T18:52:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 -- 172.21.15.104:0/2643895910 shutdown_connections 2024-02-19T18:52:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 --2- 172.21.15.104:0/2643895910 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 0x7f5aac0fcfe0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 -- 172.21.15.104:0/2643895910 >> 172.21.15.104:0/2643895910 conn(0x7f5aac0f8290 msgr2=0x7f5aac0fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 -- 172.21.15.104:0/2643895910 shutdown_connections 2024-02-19T18:52:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 -- 172.21.15.104:0/2643895910 wait complete. 2024-02-19T18:52:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 Processor -- start 2024-02-19T18:52:15.959 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 -- start start 2024-02-19T18:52:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.942+0000 7f5ab4344700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 0x7f5aac071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.943+0000 7f5ab4344700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aac0702e0 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.943+0000 7f5ab20e0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 0x7f5aac071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.943+0000 7f5ab20e0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 0x7f5aac071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:41630/0 (socket says 172.21.15.104:41630) 2024-02-19T18:52:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.943+0000 7f5ab20e0700 1 -- 172.21.15.104:0/2893102996 learned_addr learned my addr 172.21.15.104:0/2893102996 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.943+0000 7f5ab20e0700 1 -- 172.21.15.104:0/2893102996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa4009940 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.943+0000 7f5ab20e0700 1 --2- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 0x7f5aac071e60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4015040 tx=0x7f5aa4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:15.961 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.944+0000 7f5aaaffd700 1 -- 172.21.15.104:0/2893102996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa4007c80 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.944+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aac0704e0 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.944+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aac0709d0 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.944+0000 7f5aaaffd700 1 -- 172.21.15.104:0/2893102996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5aa401d910 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.944+0000 7f5aaaffd700 1 -- 172.21.15.104:0/2893102996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5aa4012940 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.944+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a980052e0 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.945+0000 7f5aaaffd700 1 -- 172.21.15.104:0/2893102996 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f5aa4019040 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.945+0000 7f5aaaffd700 1 --2- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5a94036870 0x7f5a94038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.945+0000 7f5aaaffd700 1 -- 172.21.15.104:0/2893102996 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5aa4049d80 con 0x7f5aac0fcbd0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.945+0000 7f5ab18df700 1 --2- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5a94036870 0x7f5a94038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:15.962 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.945+0000 7f5ab18df700 1 --2- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5a94036870 0x7f5a94038d20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009990 tx=0x7f5aa0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:15.966 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:15.950+0000 7f5aaaffd700 1 -- 172.21.15.104:0/2893102996 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f5aa4011b60 con 0x7f5aac0fcbd0 2024-02-19T18:52:16.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.209+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a98005590 con 0x7f5aac0fcbd0 2024-02-19T18:52:16.226 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.210+0000 7f5aaaffd700 1 -- 172.21.15.104:0/2893102996 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f5aa4017020 con 0x7f5aac0fcbd0 2024-02-19T18:52:16.226 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:16.227 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:16.228 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5a94036870 msgr2=0x7f5a94038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 --2- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5a94036870 0x7f5a94038d20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0009990 tx=0x7f5aa0006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 msgr2=0x7f5aac071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 --2- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 0x7f5aac071e60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4015040 tx=0x7f5aa4004d10 comp rx=0 tx=0).stop 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 shutdown_connections 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 --2- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5a94036870 0x7f5a94038d20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 --2- 172.21.15.104:0/2893102996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5aac0fcbd0 0x7f5aac071e60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 >> 172.21.15.104:0/2893102996 conn(0x7f5aac0f8290 msgr2=0x7f5aac0f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 shutdown_connections 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:16.212+0000 7f5ab4344700 1 -- 172.21.15.104:0/2893102996 wait complete. 2024-02-19T18:52:16.229 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:16.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:16 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2893102996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:17.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:17 smithi077 ceph-mon[29247]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:17.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:17.784 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:18.025 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:18.073 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:19.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:18 smithi077 ceph-mon[29247]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.329+0000 7fb30c7fe700 1 -- 172.21.15.104:0/3301176624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 msgr2=0x7fb3040ff7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.329+0000 7fb30c7fe700 1 --2- 172.21.15.104:0/3301176624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 0x7fb3040ff7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3000048c0 tx=0x7fb300009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.329+0000 7fb30c7fe700 1 -- 172.21.15.104:0/3301176624 shutdown_connections 2024-02-19T18:52:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.329+0000 7fb30c7fe700 1 --2- 172.21.15.104:0/3301176624 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 0x7fb3040ff7b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:19.346 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.329+0000 7fb30c7fe700 1 -- 172.21.15.104:0/3301176624 >> 172.21.15.104:0/3301176624 conn(0x7fb3040f71a0 msgr2=0x7fb3040f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.329+0000 7fb30c7fe700 1 -- 172.21.15.104:0/3301176624 shutdown_connections 2024-02-19T18:52:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.329+0000 7fb30c7fe700 1 -- 172.21.15.104:0/3301176624 wait complete. 2024-02-19T18:52:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.330+0000 7fb30c7fe700 1 Processor -- start 2024-02-19T18:52:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.330+0000 7fb30c7fe700 1 -- start start 2024-02-19T18:52:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.330+0000 7fb30c7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 0x7fb30410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.330+0000 7fb30c7fe700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30410d400 con 0x7fb304066840 2024-02-19T18:52:19.347 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.330+0000 7fb30a59a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 0x7fb30410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:19.348 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.330+0000 7fb30a59a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 0x7fb30410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:60024/0 (socket says 172.21.15.104:60024) 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.330+0000 7fb30a59a700 1 -- 172.21.15.104:0/1096391136 learned_addr learned my addr 172.21.15.104:0/1096391136 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.331+0000 7fb30a59a700 1 -- 172.21.15.104:0/1096391136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb300005040 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.331+0000 7fb30a59a700 1 --2- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 0x7fb30410cec0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb300009680 tx=0x7fb300009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.331+0000 7fb2faffd700 1 -- 172.21.15.104:0/1096391136 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb30000b5c0 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.331+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30410d600 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.331+0000 7fb2faffd700 1 -- 172.21.15.104:0/1096391136 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb300004e60 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.331+0000 7fb2faffd700 1 -- 172.21.15.104:0/1096391136 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb30001c590 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.331+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30410e2a0 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.332+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30405f5c0 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.332+0000 7fb2faffd700 1 -- 172.21.15.104:0/1096391136 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fb300013a80 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.332+0000 7fb2faffd700 1 --2- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2f00368c0 0x7fb2f0038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.332+0000 7fb2faffd700 1 -- 172.21.15.104:0/1096391136 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb300049af0 con 0x7fb304066840 2024-02-19T18:52:19.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.332+0000 7fb309d99700 1 --2- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2f00368c0 0x7fb2f0038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:19.350 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.333+0000 7fb309d99700 1 --2- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2f00368c0 0x7fb2f0038d70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009990 tx=0x7fb2f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:19.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.337+0000 7fb2faffd700 1 -- 172.21.15.104:0/1096391136 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb300025400 con 0x7fb304066840 2024-02-19T18:52:19.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.595+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb30410edf0 con 0x7fb304066840 2024-02-19T18:52:19.611 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.595+0000 7fb2faffd700 1 -- 172.21.15.104:0/1096391136 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb300011370 con 0x7fb304066840 2024-02-19T18:52:19.612 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:19.612 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:19.613 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2f00368c0 msgr2=0x7fb2f0038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 --2- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2f00368c0 0x7fb2f0038d70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009990 tx=0x7fb2f4006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 msgr2=0x7fb30410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 --2- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 0x7fb30410cec0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb300009680 tx=0x7fb300009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 shutdown_connections 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 --2- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb2f00368c0 0x7fb2f0038d70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 --2- 172.21.15.104:0/1096391136 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb304066840 0x7fb30410cec0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 >> 172.21.15.104:0/1096391136 conn(0x7fb3040f71a0 msgr2=0x7fb3040f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 shutdown_connections 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:19.597+0000 7fb30c7fe700 1 -- 172.21.15.104:0/1096391136 wait complete. 2024-02-19T18:52:19.614 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:20.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:19 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1096391136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:21.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:21.171 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:21.306 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:20 smithi077 ceph-mon[29247]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:21.419 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:21.466 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:22.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 -- 172.21.15.104:0/3503139239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 msgr2=0x7fa3600fd910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:22.975 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 --2- 172.21.15.104:0/3503139239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 0x7fa3600fd910 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0048c0 tx=0x7fa35c009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 -- 172.21.15.104:0/3503139239 shutdown_connections 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 --2- 172.21.15.104:0/3503139239 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 0x7fa3600fd910 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 -- 172.21.15.104:0/3503139239 >> 172.21.15.104:0/3503139239 conn(0x7fa3600f71a0 msgr2=0x7fa3600f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 -- 172.21.15.104:0/3503139239 shutdown_connections 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 -- 172.21.15.104:0/3503139239 wait complete. 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 Processor -- start 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.958+0000 7fa367985700 1 -- start start 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa367985700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 0x7fa3601134a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa367985700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3601139e0 con 0x7fa3600fb4f0 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa365721700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 0x7fa3601134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa365721700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 0x7fa3601134a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:60052/0 (socket says 172.21.15.104:60052) 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa365721700 1 -- 172.21.15.104:0/2379246566 learned_addr learned my addr 172.21.15.104:0/2379246566 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:22.976 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa365721700 1 -- 172.21.15.104:0/2379246566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa35c005040 con 0x7fa3600fb4f0 2024-02-19T18:52:22.977 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa365721700 1 --2- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 0x7fa3601134a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009680 tx=0x7fa35c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa355ffb700 1 -- 172.21.15.104:0/2379246566 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c00b5c0 con 0x7fa3600fb4f0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa360113be0 con 0x7fa3600fb4f0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa355ffb700 1 -- 172.21.15.104:0/2379246566 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa35c004e60 con 0x7fa3600fb4f0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.959+0000 7fa355ffb700 1 -- 172.21.15.104:0/2379246566 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa35c01c590 con 0x7fa3600fb4f0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.960+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa360114880 con 0x7fa3600fb4f0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.961+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa36005f5c0 con 0x7fa3600fb4f0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.961+0000 7fa355ffb700 1 -- 172.21.15.104:0/2379246566 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fa35c013a80 con 0x7fa3600fb4f0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.961+0000 7fa355ffb700 1 --2- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa34c036870 0x7fa34c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.961+0000 7fa355ffb700 1 -- 172.21.15.104:0/2379246566 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa35c049be0 con 0x7fa3600fb4f0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.961+0000 7fa364f20700 1 --2- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa34c036870 0x7fa34c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:22.978 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.961+0000 7fa364f20700 1 --2- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa34c036870 0x7fa34c038d20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa350009990 tx=0x7fa350006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:22.982 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:22.966+0000 7fa355ffb700 1 -- 172.21.15.104:0/2379246566 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa35c025400 con 0x7fa3600fb4f0 2024-02-19T18:52:23.236 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.220+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3601153d0 con 0x7fa3600fb4f0 2024-02-19T18:52:23.237 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.220+0000 7fa355ffb700 1 -- 172.21.15.104:0/2379246566 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fa35c011370 con 0x7fa3600fb4f0 2024-02-19T18:52:23.237 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:23.237 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:23.238 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa34c036870 msgr2=0x7fa34c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 --2- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa34c036870 0x7fa34c038d20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa350009990 tx=0x7fa350006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 msgr2=0x7fa3601134a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 --2- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 0x7fa3601134a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fa35c009680 tx=0x7fa35c009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 shutdown_connections 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 --2- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fa34c036870 0x7fa34c038d20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 --2- 172.21.15.104:0/2379246566 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa3600fb4f0 0x7fa3601134a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 >> 172.21.15.104:0/2379246566 conn(0x7fa3600f71a0 msgr2=0x7fa3600f7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 shutdown_connections 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:23.222+0000 7fa367985700 1 -- 172.21.15.104:0/2379246566 wait complete. 2024-02-19T18:52:23.239 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:23.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:23 smithi077 ceph-mon[29247]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:23.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:23 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2379246566' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:24.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:24.773 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:25.019 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:25.065 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:25.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:25 smithi077 ceph-mon[29247]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:26.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.704+0000 7f3af9d41700 1 -- 172.21.15.104:0/3156996869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 msgr2=0x7f3af40ffb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:26.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.704+0000 7f3af9d41700 1 --2- 172.21.15.104:0/3156996869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 0x7f3af40ffb00 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3ae40048c0 tx=0x7f3ae4009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:26.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.704+0000 7f3af9d41700 1 -- 172.21.15.104:0/3156996869 shutdown_connections 2024-02-19T18:52:26.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.704+0000 7f3af9d41700 1 --2- 172.21.15.104:0/3156996869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 0x7f3af40ffb00 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:26.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.704+0000 7f3af9d41700 1 -- 172.21.15.104:0/3156996869 >> 172.21.15.104:0/3156996869 conn(0x7f3af40f71a0 msgr2=0x7f3af40f95d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:26.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.704+0000 7f3af9d41700 1 -- 172.21.15.104:0/3156996869 shutdown_connections 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.704+0000 7f3af9d41700 1 -- 172.21.15.104:0/3156996869 wait complete. 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.704+0000 7f3af9d41700 1 Processor -- start 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3af9d41700 1 -- start start 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3af9d41700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 0x7f3af410ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3af9d41700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af410d3f0 con 0x7f3af40fd720 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3af37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 0x7f3af410ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3af37fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 0x7f3af410ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:48764/0 (socket says 172.21.15.104:48764) 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3af37fe700 1 -- 172.21.15.104:0/1801713622 learned_addr learned my addr 172.21.15.104:0/1801713622 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3af37fe700 1 -- 172.21.15.104:0/1801713622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae4005040 con 0x7f3af40fd720 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3af37fe700 1 --2- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 0x7f3af410ceb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009b80 tx=0x7f3ae4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.705+0000 7f3adbfff700 1 -- 172.21.15.104:0/1801713622 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae400b5c0 con 0x7f3af40fd720 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.706+0000 7f3adbfff700 1 -- 172.21.15.104:0/1801713622 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3ae400be70 con 0x7f3af40fd720 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.706+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af410d5f0 con 0x7f3af40fd720 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.706+0000 7f3adbfff700 1 -- 172.21.15.104:0/1801713622 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae401c660 con 0x7f3af40fd720 2024-02-19T18:52:26.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.706+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af410e290 con 0x7f3af40fd720 2024-02-19T18:52:26.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.706+0000 7f3adbfff700 1 -- 172.21.15.104:0/1801713622 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f3ae401c8d0 con 0x7f3af40fd720 2024-02-19T18:52:26.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.706+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af405f5c0 con 0x7f3af40fd720 2024-02-19T18:52:26.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.707+0000 7f3adbfff700 1 --2- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3adc0368c0 0x7f3adc038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:26.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.707+0000 7f3adbfff700 1 -- 172.21.15.104:0/1801713622 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3ae4049ab0 con 0x7f3af40fd720 2024-02-19T18:52:26.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.707+0000 7f3af2ffd700 1 --2- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3adc0368c0 0x7f3adc038d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:26.725 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.707+0000 7f3af2ffd700 1 --2- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3adc0368c0 0x7f3adc038d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009990 tx=0x7f3ae8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:26.727 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.711+0000 7f3adbfff700 1 -- 172.21.15.104:0/1801713622 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3ae4011db0 con 0x7f3af40fd720 2024-02-19T18:52:27.000 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.984+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3af410ede0 con 0x7f3af40fd720 2024-02-19T18:52:27.001 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.984+0000 7f3adbfff700 1 -- 172.21.15.104:0/1801713622 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f3ae4023990 con 0x7f3af40fd720 2024-02-19T18:52:27.001 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:27.001 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.986+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3adc0368c0 msgr2=0x7f3adc038d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.986+0000 7f3af9d41700 1 --2- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3adc0368c0 0x7f3adc038d70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009990 tx=0x7f3ae8006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.986+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 msgr2=0x7f3af410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.986+0000 7f3af9d41700 1 --2- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 0x7f3af410ceb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4009b80 tx=0x7f3ae4009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.986+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 shutdown_connections 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.986+0000 7f3af9d41700 1 --2- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f3adc0368c0 0x7f3adc038d70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.986+0000 7f3af9d41700 1 --2- 172.21.15.104:0/1801713622 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3af40fd720 0x7f3af410ceb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.986+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 >> 172.21.15.104:0/1801713622 conn(0x7f3af40f71a0 msgr2=0x7f3af40f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.987+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 shutdown_connections 2024-02-19T18:52:27.003 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:26.987+0000 7f3af9d41700 1 -- 172.21.15.104:0/1801713622 wait complete. 2024-02-19T18:52:27.004 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:27.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:27 smithi077 ceph-mon[29247]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:27.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:27 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1801713622' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:28.433 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:28.434 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:28.675 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:28.721 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:29.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:28 smithi077 ceph-mon[29247]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.119+0000 7fee4fea3700 1 -- 172.21.15.104:0/3299812561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 msgr2=0x7fee480ffae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.119+0000 7fee4fea3700 1 --2- 172.21.15.104:0/3299812561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 0x7fee480ffae0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fee440048c0 tx=0x7fee44009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.119+0000 7fee4fea3700 1 -- 172.21.15.104:0/3299812561 shutdown_connections 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.119+0000 7fee4fea3700 1 --2- 172.21.15.104:0/3299812561 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 0x7fee480ffae0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.119+0000 7fee4fea3700 1 -- 172.21.15.104:0/3299812561 >> 172.21.15.104:0/3299812561 conn(0x7fee480f71a0 msgr2=0x7fee480f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.119+0000 7fee4fea3700 1 -- 172.21.15.104:0/3299812561 shutdown_connections 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.119+0000 7fee4fea3700 1 -- 172.21.15.104:0/3299812561 wait complete. 2024-02-19T18:52:30.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.120+0000 7fee4fea3700 1 Processor -- start 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.120+0000 7fee4fea3700 1 -- start start 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.120+0000 7fee4fea3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 0x7fee4810cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.120+0000 7fee4fea3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4810d3e0 con 0x7fee480fd700 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.120+0000 7fee4dc3f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 0x7fee4810cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.120+0000 7fee4dc3f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 0x7fee4810cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:48782/0 (socket says 172.21.15.104:48782) 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.120+0000 7fee4dc3f700 1 -- 172.21.15.104:0/2203241324 learned_addr learned my addr 172.21.15.104:0/2203241324 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.120+0000 7fee4dc3f700 1 -- 172.21.15.104:0/2203241324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee44005040 con 0x7fee480fd700 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.121+0000 7fee4dc3f700 1 --2- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 0x7fee4810cea0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fee44009b80 tx=0x7fee44009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.121+0000 7fee3e7fc700 1 -- 172.21.15.104:0/2203241324 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4400b5c0 con 0x7fee480fd700 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.121+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4810d5e0 con 0x7fee480fd700 2024-02-19T18:52:30.138 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.121+0000 7fee3e7fc700 1 -- 172.21.15.104:0/2203241324 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fee4400be70 con 0x7fee480fd700 2024-02-19T18:52:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.121+0000 7fee3e7fc700 1 -- 172.21.15.104:0/2203241324 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee4401c660 con 0x7fee480fd700 2024-02-19T18:52:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.121+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4810e280 con 0x7fee480fd700 2024-02-19T18:52:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.122+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4805f5c0 con 0x7fee480fd700 2024-02-19T18:52:30.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.122+0000 7fee3e7fc700 1 -- 172.21.15.104:0/2203241324 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fee44004d30 con 0x7fee480fd700 2024-02-19T18:52:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.122+0000 7fee3e7fc700 1 --2- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fee34036870 0x7fee34038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.122+0000 7fee3e7fc700 1 -- 172.21.15.104:0/2203241324 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fee44049a30 con 0x7fee480fd700 2024-02-19T18:52:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.122+0000 7fee4d43e700 1 --2- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fee34036870 0x7fee34038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:30.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.123+0000 7fee4d43e700 1 --2- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fee34036870 0x7fee34038d20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee38009990 tx=0x7fee38006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:30.144 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.127+0000 7fee3e7fc700 1 -- 172.21.15.104:0/2203241324 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fee44025400 con 0x7fee480fd700 2024-02-19T18:52:30.402 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.386+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee4810edd0 con 0x7fee480fd700 2024-02-19T18:52:30.403 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.387+0000 7fee3e7fc700 1 -- 172.21.15.104:0/2203241324 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fee44023300 con 0x7fee480fd700 2024-02-19T18:52:30.404 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:30.404 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:30.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fee34036870 msgr2=0x7fee34038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:30.405 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 --2- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fee34036870 0x7fee34038d20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fee38009990 tx=0x7fee38006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 msgr2=0x7fee4810cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 --2- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 0x7fee4810cea0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fee44009b80 tx=0x7fee44009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 shutdown_connections 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 --2- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fee34036870 0x7fee34038d20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 --2- 172.21.15.104:0/2203241324 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fee480fd700 0x7fee4810cea0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 >> 172.21.15.104:0/2203241324 conn(0x7fee480f71a0 msgr2=0x7fee480f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 shutdown_connections 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:30.389+0000 7fee4fea3700 1 -- 172.21.15.104:0/2203241324 wait complete. 2024-02-19T18:52:30.406 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:31.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:31 smithi077 ceph-mon[29247]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:31.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:31 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2203241324' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:31.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:31.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:32.242 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:32.290 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:33.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:33 smithi077 ceph-mon[29247]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:33.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.542+0000 7fd47acbc700 1 -- 172.21.15.104:0/526609833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 msgr2=0x7fd4740fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:33.559 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.542+0000 7fd47acbc700 1 --2- 172.21.15.104:0/526609833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 0x7fd4740fbed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd4680048c0 tx=0x7fd468009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 -- 172.21.15.104:0/526609833 shutdown_connections 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 --2- 172.21.15.104:0/526609833 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 0x7fd4740fbed0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 -- 172.21.15.104:0/526609833 >> 172.21.15.104:0/526609833 conn(0x7fd4740f71a0 msgr2=0x7fd4740f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 -- 172.21.15.104:0/526609833 shutdown_connections 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 -- 172.21.15.104:0/526609833 wait complete. 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 Processor -- start 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 -- start start 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 0x7fd47410fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:33.560 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.543+0000 7fd47acbc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd474066830 con 0x7fd4740fbac0 2024-02-19T18:52:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd478a58700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 0x7fd47410fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd478a58700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 0x7fd47410fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:48806/0 (socket says 172.21.15.104:48806) 2024-02-19T18:52:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd478a58700 1 -- 172.21.15.104:0/2198651739 learned_addr learned my addr 172.21.15.104:0/2198651739 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd478a58700 1 -- 172.21.15.104:0/2198651739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd468005040 con 0x7fd4740fbac0 2024-02-19T18:52:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd478a58700 1 --2- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 0x7fd47410fbe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd468009680 tx=0x7fd468009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd4717fa700 1 -- 172.21.15.104:0/2198651739 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd46800bed0 con 0x7fd4740fbac0 2024-02-19T18:52:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd4717fa700 1 -- 172.21.15.104:0/2198651739 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd46800b5c0 con 0x7fd4740fbac0 2024-02-19T18:52:33.561 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd474066a30 con 0x7fd4740fbac0 2024-02-19T18:52:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd4717fa700 1 -- 172.21.15.104:0/2198651739 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd46801c5b0 con 0x7fd4740fbac0 2024-02-19T18:52:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.544+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd474110360 con 0x7fd4740fbac0 2024-02-19T18:52:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.545+0000 7fd4717fa700 1 -- 172.21.15.104:0/2198651739 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fd468004d30 con 0x7fd4740fbac0 2024-02-19T18:52:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.545+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474105890 con 0x7fd4740fbac0 2024-02-19T18:52:33.562 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.545+0000 7fd4717fa700 1 --2- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd464036820 0x7fd464038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:33.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.546+0000 7fd4717fa700 1 -- 172.21.15.104:0/2198651739 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd468049b70 con 0x7fd4740fbac0 2024-02-19T18:52:33.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.546+0000 7fd473fff700 1 --2- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd464036820 0x7fd464038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:33.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.546+0000 7fd473fff700 1 --2- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd464036820 0x7fd464038cd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd460009990 tx=0x7fd460006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:33.567 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.551+0000 7fd4717fa700 1 -- 172.21.15.104:0/2198651739 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd468011940 con 0x7fd4740fbac0 2024-02-19T18:52:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.805+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd47404bed0 con 0x7fd4740fbac0 2024-02-19T18:52:33.822 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.806+0000 7fd4717fa700 1 -- 172.21.15.104:0/2198651739 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd468011370 con 0x7fd4740fbac0 2024-02-19T18:52:33.822 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:33.823 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd464036820 msgr2=0x7fd464038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 --2- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd464036820 0x7fd464038cd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd460009990 tx=0x7fd460006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 msgr2=0x7fd47410fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 --2- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 0x7fd47410fbe0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd468009680 tx=0x7fd468009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 shutdown_connections 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 --2- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd464036820 0x7fd464038cd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 --2- 172.21.15.104:0/2198651739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd4740fbac0 0x7fd47410fbe0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 >> 172.21.15.104:0/2198651739 conn(0x7fd4740f71a0 msgr2=0x7fd4740f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:33.824 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 shutdown_connections 2024-02-19T18:52:33.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:33.808+0000 7fd47acbc700 1 -- 172.21.15.104:0/2198651739 wait complete. 2024-02-19T18:52:33.825 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:34.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:34 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2198651739' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:35.324 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:35.325 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:35.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:35 smithi077 ceph-mon[29247]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:35.571 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:35.617 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.886+0000 7f9888a28700 1 -- 172.21.15.104:0/2339827516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 msgr2=0x7f98840fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.886+0000 7f9888a28700 1 --2- 172.21.15.104:0/2339827516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 0x7f98840fcfe0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f98780048c0 tx=0x7f9878009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.886+0000 7f9888a28700 1 -- 172.21.15.104:0/2339827516 shutdown_connections 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.886+0000 7f9888a28700 1 --2- 172.21.15.104:0/2339827516 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 0x7f98840fcfe0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.886+0000 7f9888a28700 1 -- 172.21.15.104:0/2339827516 >> 172.21.15.104:0/2339827516 conn(0x7f98840f8270 msgr2=0x7f98840fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.886+0000 7f9888a28700 1 -- 172.21.15.104:0/2339827516 shutdown_connections 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.886+0000 7f9888a28700 1 -- 172.21.15.104:0/2339827516 wait complete. 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.887+0000 7f9888a28700 1 Processor -- start 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.887+0000 7f9888a28700 1 -- start start 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.887+0000 7f9888a28700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 0x7f988410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:36.904 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.887+0000 7f9888a28700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988410d400 con 0x7f98840fcbd0 2024-02-19T18:52:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.887+0000 7f988259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 0x7f988410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.887+0000 7f988259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 0x7f988410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:33618/0 (socket says 172.21.15.104:33618) 2024-02-19T18:52:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.887+0000 7f988259c700 1 -- 172.21.15.104:0/3332746547 learned_addr learned my addr 172.21.15.104:0/3332746547 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.887+0000 7f988259c700 1 -- 172.21.15.104:0/3332746547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9878005040 con 0x7f98840fcbd0 2024-02-19T18:52:36.905 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.888+0000 7f988259c700 1 --2- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 0x7f988410cec0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9878009680 tx=0x7f9878009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.888+0000 7f9872ffd700 1 -- 172.21.15.104:0/3332746547 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f987800bed0 con 0x7f98840fcbd0 2024-02-19T18:52:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.888+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988410d600 con 0x7f98840fcbd0 2024-02-19T18:52:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.888+0000 7f9872ffd700 1 -- 172.21.15.104:0/3332746547 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f987800b5c0 con 0x7f98840fcbd0 2024-02-19T18:52:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.888+0000 7f9872ffd700 1 -- 172.21.15.104:0/3332746547 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f987801c5b0 con 0x7f98840fcbd0 2024-02-19T18:52:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.888+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988410e2a0 con 0x7f98840fcbd0 2024-02-19T18:52:36.906 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.889+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988405f5c0 con 0x7f98840fcbd0 2024-02-19T18:52:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.889+0000 7f9872ffd700 1 -- 172.21.15.104:0/3332746547 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f9878013a80 con 0x7f98840fcbd0 2024-02-19T18:52:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.889+0000 7f9872ffd700 1 --2- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f986c036820 0x7f986c038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.889+0000 7f9872ffd700 1 -- 172.21.15.104:0/3332746547 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9878049b70 con 0x7f98840fcbd0 2024-02-19T18:52:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.889+0000 7f9881d9b700 1 --2- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f986c036820 0x7f986c038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:36.907 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.890+0000 7f9881d9b700 1 --2- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f986c036820 0x7f986c038cd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9874009990 tx=0x7f9874006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:36.910 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:36.894+0000 7f9872ffd700 1 -- 172.21.15.104:0/3332746547 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9878011940 con 0x7f98840fcbd0 2024-02-19T18:52:37.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.148+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f988410edf0 con 0x7f98840fcbd0 2024-02-19T18:52:37.165 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.149+0000 7f9872ffd700 1 -- 172.21.15.104:0/3332746547 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9878011370 con 0x7f98840fcbd0 2024-02-19T18:52:37.166 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:37.166 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:37.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f986c036820 msgr2=0x7f986c038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:37.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 --2- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f986c036820 0x7f986c038cd0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9874009990 tx=0x7f9874006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:37.167 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 msgr2=0x7f988410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:37.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 --2- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 0x7f988410cec0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9878009680 tx=0x7f9878009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:37.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 shutdown_connections 2024-02-19T18:52:37.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 --2- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f986c036820 0x7f986c038cd0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:37.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 --2- 172.21.15.104:0/3332746547 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98840fcbd0 0x7f988410cec0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:37.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 >> 172.21.15.104:0/3332746547 conn(0x7f98840f8270 msgr2=0x7f98840f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:37.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 shutdown_connections 2024-02-19T18:52:37.168 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:37.151+0000 7f9888a28700 1 -- 172.21.15.104:0/3332746547 wait complete. 2024-02-19T18:52:37.168 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:37.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:37 smithi077 ceph-mon[29247]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:37.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:37 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3332746547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:38.753 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:38.753 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:39.000 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:39.045 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:39.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:38 smithi077 ceph-mon[29247]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:40.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 -- 172.21.15.104:0/988378400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82480fbac0 msgr2=0x7f82480fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:40.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 --2- 172.21.15.104:0/988378400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82480fbac0 0x7f82480fbed0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f82380048c0 tx=0x7f8238009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:40.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 -- 172.21.15.104:0/988378400 shutdown_connections 2024-02-19T18:52:40.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 --2- 172.21.15.104:0/988378400 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f82480fbac0 0x7f82480fbed0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:40.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 -- 172.21.15.104:0/988378400 >> 172.21.15.104:0/988378400 conn(0x7f82480f71a0 msgr2=0x7f82480f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 -- 172.21.15.104:0/988378400 shutdown_connections 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 -- 172.21.15.104:0/988378400 wait complete. 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 Processor -- start 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.335+0000 7f824d1d8700 1 -- start start 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f824d1d8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f824810d010 0x7f824810d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f824d1d8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824810d960 con 0x7f824810d010 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f8246d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f824810d010 0x7f824810d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f8246d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f824810d010 0x7f824810d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:33640/0 (socket says 172.21.15.104:33640) 2024-02-19T18:52:40.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f8246d9d700 1 -- 172.21.15.104:0/1346485371 learned_addr learned my addr 172.21.15.104:0/1346485371 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f8246d9d700 1 -- 172.21.15.104:0/1346485371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8238005040 con 0x7f824810d010 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f8246d9d700 1 --2- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f824810d010 0x7f824810d420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8238009040 tx=0x7f8238009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f822f7fe700 1 -- 172.21.15.104:0/1346485371 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f823800b5c0 con 0x7f824810d010 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.336+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824810db60 con 0x7f824810d010 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.337+0000 7f822f7fe700 1 -- 172.21.15.104:0/1346485371 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8238004e60 con 0x7f824810d010 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.337+0000 7f822f7fe700 1 -- 172.21.15.104:0/1346485371 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f823801c5b0 con 0x7f824810d010 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.337+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824810e670 con 0x7f824810d010 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.337+0000 7f822f7fe700 1 -- 172.21.15.104:0/1346485371 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f8238013420 con 0x7f824810d010 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.337+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824804bed0 con 0x7f824810d010 2024-02-19T18:52:40.355 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.338+0000 7f822f7fe700 1 --2- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8230036870 0x7f8230038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:40.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.338+0000 7f822f7fe700 1 -- 172.21.15.104:0/1346485371 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8238049a30 con 0x7f824810d010 2024-02-19T18:52:40.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.338+0000 7f824659c700 1 --2- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8230036870 0x7f8230038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:40.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.338+0000 7f824659c700 1 --2- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8230036870 0x7f8230038d20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f823c009990 tx=0x7f823c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:40.359 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.343+0000 7f822f7fe700 1 -- 172.21.15.104:0/1346485371 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8238025400 con 0x7f824810d010 2024-02-19T18:52:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.601+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8248101880 con 0x7f824810d010 2024-02-19T18:52:40.618 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.601+0000 7f822f7fe700 1 -- 172.21.15.104:0/1346485371 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f8238011370 con 0x7f824810d010 2024-02-19T18:52:40.618 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:40.618 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8230036870 msgr2=0x7f8230038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 --2- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8230036870 0x7f8230038d20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f823c009990 tx=0x7f823c006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f824810d010 msgr2=0x7f824810d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 --2- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f824810d010 0x7f824810d420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8238009040 tx=0x7f8238009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 shutdown_connections 2024-02-19T18:52:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 --2- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f8230036870 0x7f8230038d20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 --2- 172.21.15.104:0/1346485371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f824810d010 0x7f824810d420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:40.620 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 >> 172.21.15.104:0/1346485371 conn(0x7f82480f71a0 msgr2=0x7f8248066410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:40.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 shutdown_connections 2024-02-19T18:52:40.621 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:40.603+0000 7f824d1d8700 1 -- 172.21.15.104:0/1346485371 wait complete. 2024-02-19T18:52:40.621 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:41.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:41 smithi077 ceph-mon[29247]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:41.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:41 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1346485371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:42.157 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:42.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:42.403 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:42.450 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:43.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:43 smithi077 ceph-mon[29247]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:43.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 -- 172.21.15.104:0/3093515126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 msgr2=0x7f89d00fcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:43.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 --2- 172.21.15.104:0/3093515126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 0x7f89d00fcfd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f89c00048c0 tx=0x7f89c0009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:43.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 -- 172.21.15.104:0/3093515126 shutdown_connections 2024-02-19T18:52:43.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 --2- 172.21.15.104:0/3093515126 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 0x7f89d00fcfd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:43.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 -- 172.21.15.104:0/3093515126 >> 172.21.15.104:0/3093515126 conn(0x7f89d00f82a0 msgr2=0x7f89d00fa6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:43.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 -- 172.21.15.104:0/3093515126 shutdown_connections 2024-02-19T18:52:43.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 -- 172.21.15.104:0/3093515126 wait complete. 2024-02-19T18:52:43.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 Processor -- start 2024-02-19T18:52:43.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.839+0000 7f89d7c91700 1 -- start start 2024-02-19T18:52:43.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.840+0000 7f89d7c91700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 0x7f89d010cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:43.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.840+0000 7f89d7c91700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d010d420 con 0x7f89d00fcbc0 2024-02-19T18:52:43.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.840+0000 7f89d5a2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 0x7f89d010cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:43.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.840+0000 7f89d5a2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 0x7f89d010cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:33656/0 (socket says 172.21.15.104:33656) 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.840+0000 7f89d5a2d700 1 -- 172.21.15.104:0/3117279187 learned_addr learned my addr 172.21.15.104:0/3117279187 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.840+0000 7f89d5a2d700 1 -- 172.21.15.104:0/3117279187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c0005040 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.841+0000 7f89d5a2d700 1 --2- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 0x7f89d010cee0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009b80 tx=0x7f89c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.841+0000 7f89c67fc700 1 -- 172.21.15.104:0/3117279187 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c000b5c0 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.841+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d010d620 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.841+0000 7f89c67fc700 1 -- 172.21.15.104:0/3117279187 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f89c000be70 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.841+0000 7f89c67fc700 1 -- 172.21.15.104:0/3117279187 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89c001c660 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.841+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d010e2c0 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.842+0000 7f89c67fc700 1 -- 172.21.15.104:0/3117279187 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f89c0013420 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.842+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d005f5c0 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.842+0000 7f89c67fc700 1 --2- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f89bc036870 0x7f89bc038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.842+0000 7f89c67fc700 1 -- 172.21.15.104:0/3117279187 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f89c00499c0 con 0x7f89d00fcbc0 2024-02-19T18:52:43.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.842+0000 7f89d522c700 1 --2- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f89bc036870 0x7f89bc038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:43.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.843+0000 7f89d522c700 1 --2- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f89bc036870 0x7f89bc038d20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009990 tx=0x7f89cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:43.863 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:43.847+0000 7f89c67fc700 1 -- 172.21.15.104:0/3117279187 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f89c0011550 con 0x7f89d00fcbc0 2024-02-19T18:52:44.118 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.102+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89d010ee10 con 0x7f89d00fcbc0 2024-02-19T18:52:44.119 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.102+0000 7f89c67fc700 1 -- 172.21.15.104:0/3117279187 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f89c0011370 con 0x7f89d00fcbc0 2024-02-19T18:52:44.119 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:44.119 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:44.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f89bc036870 msgr2=0x7f89bc038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:44.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 --2- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f89bc036870 0x7f89bc038d20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f89cc009990 tx=0x7f89cc006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:44.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 msgr2=0x7f89d010cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:44.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 --2- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 0x7f89d010cee0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f89c0009b80 tx=0x7f89c0009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:44.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 shutdown_connections 2024-02-19T18:52:44.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 --2- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f89bc036870 0x7f89bc038d20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:44.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 --2- 172.21.15.104:0/3117279187 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f89d00fcbc0 0x7f89d010cee0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:44.121 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 >> 172.21.15.104:0/3117279187 conn(0x7f89d00f82a0 msgr2=0x7f89d00f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:44.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 shutdown_connections 2024-02-19T18:52:44.122 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:44.104+0000 7f89d7c91700 1 -- 172.21.15.104:0/3117279187 wait complete. 2024-02-19T18:52:44.122 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:44.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:44 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3117279187' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:45.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:45 smithi077 ceph-mon[29247]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:45.718 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:45.718 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:45.964 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:46.011 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:47.382 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.364+0000 7ff70c13c700 1 -- 172.21.15.104:0/1101810218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 msgr2=0x7ff7040fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.364+0000 7ff70c13c700 1 --2- 172.21.15.104:0/1101810218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 0x7ff7040fbed0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff7000048c0 tx=0x7ff700009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.364+0000 7ff70c13c700 1 -- 172.21.15.104:0/1101810218 shutdown_connections 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.364+0000 7ff70c13c700 1 --2- 172.21.15.104:0/1101810218 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 0x7ff7040fbed0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.364+0000 7ff70c13c700 1 -- 172.21.15.104:0/1101810218 >> 172.21.15.104:0/1101810218 conn(0x7ff7040f71a0 msgr2=0x7ff7040f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.364+0000 7ff70c13c700 1 -- 172.21.15.104:0/1101810218 shutdown_connections 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.364+0000 7ff70c13c700 1 -- 172.21.15.104:0/1101810218 wait complete. 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.365+0000 7ff70c13c700 1 Processor -- start 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.365+0000 7ff70c13c700 1 -- start start 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.365+0000 7ff70c13c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 0x7ff70410fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.365+0000 7ff70c13c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff704066830 con 0x7ff7040fbac0 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.365+0000 7ff709ed8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 0x7ff70410fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.365+0000 7ff709ed8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 0x7ff70410fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:47376/0 (socket says 172.21.15.104:47376) 2024-02-19T18:52:47.383 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.365+0000 7ff709ed8700 1 -- 172.21.15.104:0/2654371105 learned_addr learned my addr 172.21.15.104:0/2654371105 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:47.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.366+0000 7ff709ed8700 1 -- 172.21.15.104:0/2654371105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff700005040 con 0x7ff7040fbac0 2024-02-19T18:52:47.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.366+0000 7ff709ed8700 1 --2- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 0x7ff70410fbe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff700009b80 tx=0x7ff700009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:47.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.366+0000 7ff6fa7fc700 1 -- 172.21.15.104:0/2654371105 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff70000b5c0 con 0x7ff7040fbac0 2024-02-19T18:52:47.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.366+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff704066a30 con 0x7ff7040fbac0 2024-02-19T18:52:47.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.366+0000 7ff6fa7fc700 1 -- 172.21.15.104:0/2654371105 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff70000be70 con 0x7ff7040fbac0 2024-02-19T18:52:47.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.366+0000 7ff6fa7fc700 1 -- 172.21.15.104:0/2654371105 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff70001c660 con 0x7ff7040fbac0 2024-02-19T18:52:47.384 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.366+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff704110360 con 0x7ff7040fbac0 2024-02-19T18:52:47.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.367+0000 7ff6fa7fc700 1 -- 172.21.15.104:0/2654371105 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ff700004d30 con 0x7ff7040fbac0 2024-02-19T18:52:47.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.367+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff704105890 con 0x7ff7040fbac0 2024-02-19T18:52:47.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.367+0000 7ff6fa7fc700 1 --2- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff6f0036870 0x7ff6f0038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:47.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.367+0000 7ff6fa7fc700 1 -- 172.21.15.104:0/2654371105 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff700049a00 con 0x7ff7040fbac0 2024-02-19T18:52:47.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.367+0000 7ff7096d7700 1 --2- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff6f0036870 0x7ff6f0038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:47.385 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.368+0000 7ff7096d7700 1 --2- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff6f0036870 0x7ff6f0038d20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009990 tx=0x7ff6f4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:47.389 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.372+0000 7ff6fa7fc700 1 -- 172.21.15.104:0/2654371105 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff700025400 con 0x7ff7040fbac0 2024-02-19T18:52:47.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:47 smithi077 ceph-mon[29247]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:47.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.629+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff70404bed0 con 0x7ff7040fbac0 2024-02-19T18:52:47.646 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.630+0000 7ff6fa7fc700 1 -- 172.21.15.104:0/2654371105 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff700011370 con 0x7ff7040fbac0 2024-02-19T18:52:47.647 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:47.647 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff6f0036870 msgr2=0x7ff6f0038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 --2- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff6f0036870 0x7ff6f0038d20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009990 tx=0x7ff6f4006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 msgr2=0x7ff70410fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 --2- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 0x7ff70410fbe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff700009b80 tx=0x7ff700009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 shutdown_connections 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 --2- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff6f0036870 0x7ff6f0038d20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 --2- 172.21.15.104:0/2654371105 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff7040fbac0 0x7ff70410fbe0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 >> 172.21.15.104:0/2654371105 conn(0x7ff7040f71a0 msgr2=0x7ff7040f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 shutdown_connections 2024-02-19T18:52:47.649 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:47.632+0000 7ff70c13c700 1 -- 172.21.15.104:0/2654371105 wait complete. 2024-02-19T18:52:47.650 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:48.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:48 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2654371105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:49.165 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:49.165 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:49.406 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:49.451 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:49.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:49 smithi077 ceph-mon[29247]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.703+0000 7f32ea900700 1 -- 172.21.15.104:0/3970200398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 msgr2=0x7f32e40ff820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.703+0000 7f32ea900700 1 --2- 172.21.15.104:0/3970200398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 0x7f32e40ff820 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f32d80048c0 tx=0x7f32d8009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.703+0000 7f32ea900700 1 -- 172.21.15.104:0/3970200398 shutdown_connections 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.703+0000 7f32ea900700 1 --2- 172.21.15.104:0/3970200398 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 0x7f32e40ff820 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.703+0000 7f32ea900700 1 -- 172.21.15.104:0/3970200398 >> 172.21.15.104:0/3970200398 conn(0x7f32e40f7210 msgr2=0x7f32e40f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.703+0000 7f32ea900700 1 -- 172.21.15.104:0/3970200398 shutdown_connections 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.703+0000 7f32ea900700 1 -- 172.21.15.104:0/3970200398 wait complete. 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.704+0000 7f32ea900700 1 Processor -- start 2024-02-19T18:52:50.722 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.704+0000 7f32ea900700 1 -- start start 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.704+0000 7f32ea900700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 0x7f32e40fd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.704+0000 7f32ea900700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e40fb720 con 0x7f32e4066760 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32e3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 0x7f32e40fd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32e3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 0x7f32e40fd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:47394/0 (socket says 172.21.15.104:47394) 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32e3fff700 1 -- 172.21.15.104:0/1857136508 learned_addr learned my addr 172.21.15.104:0/1857136508 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32e3fff700 1 -- 172.21.15.104:0/1857136508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d8005040 con 0x7f32e4066760 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32e3fff700 1 --2- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 0x7f32e40fd2a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009680 tx=0x7f32d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32e0ff9700 1 -- 172.21.15.104:0/1857136508 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d800bed0 con 0x7f32e4066760 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32e0ff9700 1 -- 172.21.15.104:0/1857136508 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32d800b5c0 con 0x7f32e4066760 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e40fb920 con 0x7f32e4066760 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32e0ff9700 1 -- 172.21.15.104:0/1857136508 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32d801c5b0 con 0x7f32e4066760 2024-02-19T18:52:50.723 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.705+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e40fbe10 con 0x7f32e4066760 2024-02-19T18:52:50.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.706+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e41068f0 con 0x7f32e4066760 2024-02-19T18:52:50.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.706+0000 7f32e0ff9700 1 -- 172.21.15.104:0/1857136508 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f32d8013a80 con 0x7f32e4066760 2024-02-19T18:52:50.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.706+0000 7f32e0ff9700 1 --2- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f32cc036820 0x7f32cc038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:50.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.706+0000 7f32e0ff9700 1 -- 172.21.15.104:0/1857136508 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f32d8049ab0 con 0x7f32e4066760 2024-02-19T18:52:50.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.707+0000 7f32e37fe700 1 --2- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f32cc036820 0x7f32cc038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:50.724 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.707+0000 7f32e37fe700 1 --2- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f32cc036820 0x7f32cc038cd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f32d4009990 tx=0x7f32d4006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:50.728 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.711+0000 7f32e0ff9700 1 -- 172.21.15.104:0/1857136508 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f32d8011940 con 0x7f32e4066760 2024-02-19T18:52:50.983 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.966+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32e404bed0 con 0x7f32e4066760 2024-02-19T18:52:50.984 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.967+0000 7f32e0ff9700 1 -- 172.21.15.104:0/1857136508 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f32d8011370 con 0x7f32e4066760 2024-02-19T18:52:50.984 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:50.984 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:50.985 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f32cc036820 msgr2=0x7f32cc038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 --2- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f32cc036820 0x7f32cc038cd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f32d4009990 tx=0x7f32d4006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 msgr2=0x7f32e40fd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 --2- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 0x7f32e40fd2a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009680 tx=0x7f32d8009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 shutdown_connections 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 --2- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f32cc036820 0x7f32cc038cd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 --2- 172.21.15.104:0/1857136508 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f32e4066760 0x7f32e40fd2a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 >> 172.21.15.104:0/1857136508 conn(0x7f32e40f7210 msgr2=0x7f32e40f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 shutdown_connections 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:50.969+0000 7f32ea900700 1 -- 172.21.15.104:0/1857136508 wait complete. 2024-02-19T18:52:50.986 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:51.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:51 smithi077 ceph-mon[29247]: pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:51.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:51 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1857136508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:52.551 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:52.552 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:52.798 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:52.846 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:53.434 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:53 smithi077 ceph-mon[29247]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:53.434 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:53 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:52:53.434 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:53 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:52:54.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.235+0000 7f815af06700 1 -- 172.21.15.104:0/3170767015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 msgr2=0x7f81540fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.235+0000 7f815af06700 1 --2- 172.21.15.104:0/3170767015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 0x7f81540fbee0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f814c009c60 tx=0x7f814c004d10 comp rx=0 tx=0).stop 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.235+0000 7f815af06700 1 -- 172.21.15.104:0/3170767015 shutdown_connections 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.235+0000 7f815af06700 1 --2- 172.21.15.104:0/3170767015 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 0x7f81540fbee0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.235+0000 7f815af06700 1 -- 172.21.15.104:0/3170767015 >> 172.21.15.104:0/3170767015 conn(0x7f81540f7190 msgr2=0x7f81540f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.235+0000 7f815af06700 1 -- 172.21.15.104:0/3170767015 shutdown_connections 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.235+0000 7f815af06700 1 -- 172.21.15.104:0/3170767015 wait complete. 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.236+0000 7f815af06700 1 Processor -- start 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.236+0000 7f815af06700 1 -- start start 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.236+0000 7f815af06700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 0x7f815410cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.236+0000 7f815af06700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815410d400 con 0x7f81540fbad0 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.236+0000 7f8158ca2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 0x7f815410cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:54.253 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.236+0000 7f8158ca2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 0x7f815410cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:47410/0 (socket says 172.21.15.104:47410) 2024-02-19T18:52:54.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.236+0000 7f8158ca2700 1 -- 172.21.15.104:0/1375203263 learned_addr learned my addr 172.21.15.104:0/1375203263 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:54.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.236+0000 7f8158ca2700 1 -- 172.21.15.104:0/1375203263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f814c009940 con 0x7f81540fbad0 2024-02-19T18:52:54.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.237+0000 7f8158ca2700 1 --2- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 0x7f815410cec0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f814c015040 tx=0x7f814c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:54.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.237+0000 7f81517fa700 1 -- 172.21.15.104:0/1375203263 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f814c007c80 con 0x7f81540fbad0 2024-02-19T18:52:54.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.237+0000 7f81517fa700 1 -- 172.21.15.104:0/1375203263 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f814c01d910 con 0x7f81540fbad0 2024-02-19T18:52:54.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.237+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815410d600 con 0x7f81540fbad0 2024-02-19T18:52:54.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.237+0000 7f81517fa700 1 -- 172.21.15.104:0/1375203263 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f814c0129f0 con 0x7f81540fbad0 2024-02-19T18:52:54.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.237+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815410e2a0 con 0x7f81540fbad0 2024-02-19T18:52:54.255 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.238+0000 7f81517fa700 1 -- 172.21.15.104:0/1375203263 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f814c019040 con 0x7f81540fbad0 2024-02-19T18:52:54.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.238+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815405f5c0 con 0x7f81540fbad0 2024-02-19T18:52:54.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.238+0000 7f81517fa700 1 --2- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f813c036870 0x7f813c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:54.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.238+0000 7f81517fa700 1 -- 172.21.15.104:0/1375203263 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f814c04a450 con 0x7f81540fbad0 2024-02-19T18:52:54.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.238+0000 7f8153fff700 1 --2- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f813c036870 0x7f813c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:54.256 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.238+0000 7f8153fff700 1 --2- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f813c036870 0x7f813c038d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8148009990 tx=0x7f8148006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:54.263 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.247+0000 7f81517fa700 1 -- 172.21.15.104:0/1375203263 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f814c02eb40 con 0x7f81540fbad0 2024-02-19T18:52:54.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.501+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f815410e5c0 con 0x7f81540fbad0 2024-02-19T18:52:54.518 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.502+0000 7f81517fa700 1 -- 172.21.15.104:0/1375203263 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f814c017070 con 0x7f81540fbad0 2024-02-19T18:52:54.518 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:54.518 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f813c036870 msgr2=0x7f813c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 --2- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f813c036870 0x7f813c038d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8148009990 tx=0x7f8148006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 msgr2=0x7f815410cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 --2- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 0x7f815410cec0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f814c015040 tx=0x7f814c004d10 comp rx=0 tx=0).stop 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 shutdown_connections 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 --2- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f813c036870 0x7f813c038d20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 --2- 172.21.15.104:0/1375203263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f81540fbad0 0x7f815410cec0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 >> 172.21.15.104:0/1375203263 conn(0x7f81540f7190 msgr2=0x7f81540f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:54.521 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 shutdown_connections 2024-02-19T18:52:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:54.504+0000 7f815af06700 1 -- 172.21.15.104:0/1375203263 wait complete. 2024-02-19T18:52:54.522 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:54.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:54 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:52:54.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:54 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:52:54.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:54 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:52:54.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:54 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:52:54.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:54 smithi077 ceph-mon[29247]: Deploying daemon prometheus.smithi077 on smithi077 2024-02-19T18:52:55.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:55 smithi077 ceph-mon[29247]: pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:55.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:55 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1375203263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:56.036 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:56.037 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:56.281 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:56.327 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:52:57.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:57 smithi077 ceph-mon[29247]: pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:57.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:57 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:52:57.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.830+0000 7ff3f0856700 1 -- 172.21.15.104:0/3251999839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 msgr2=0x7ff3ec0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:57.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.830+0000 7ff3f0856700 1 --2- 172.21.15.104:0/3251999839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 0x7ff3ec0fbed0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff3e40048c0 tx=0x7ff3e4009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:57.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.830+0000 7ff3f0856700 1 -- 172.21.15.104:0/3251999839 shutdown_connections 2024-02-19T18:52:57.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.830+0000 7ff3f0856700 1 --2- 172.21.15.104:0/3251999839 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 0x7ff3ec0fbed0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:57.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.830+0000 7ff3f0856700 1 -- 172.21.15.104:0/3251999839 >> 172.21.15.104:0/3251999839 conn(0x7ff3ec0f71a0 msgr2=0x7ff3ec0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:57.854 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.830+0000 7ff3f0856700 1 -- 172.21.15.104:0/3251999839 shutdown_connections 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.830+0000 7ff3f0856700 1 -- 172.21.15.104:0/3251999839 wait complete. 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.830+0000 7ff3f0856700 1 Processor -- start 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3f0856700 1 -- start start 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3f0856700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 0x7ff3ec0f9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3f0856700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3ec0f7690 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3ea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 0x7ff3ec0f9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3ea59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 0x7ff3ec0f9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:51176/0 (socket says 172.21.15.104:51176) 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3ea59c700 1 -- 172.21.15.104:0/1223502958 learned_addr learned my addr 172.21.15.104:0/1223502958 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3ea59c700 1 -- 172.21.15.104:0/1223502958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4005040 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3ea59c700 1 --2- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 0x7ff3ec0f9210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009680 tx=0x7ff3e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3daffd700 1 -- 172.21.15.104:0/1223502958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3e400bed0 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3ec0f7890 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3daffd700 1 -- 172.21.15.104:0/1223502958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3e400b5c0 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3daffd700 1 -- 172.21.15.104:0/1223502958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3e401c5b0 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.831+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3ec0f7d80 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.832+0000 7ff3daffd700 1 -- 172.21.15.104:0/1223502958 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7ff3e4013420 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.832+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3ec1068a0 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.832+0000 7ff3daffd700 1 --2- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff3d4036820 0x7ff3d4038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:52:57.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.832+0000 7ff3daffd700 1 -- 172.21.15.104:0/1223502958 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff3e4049c10 con 0x7ff3ec0fbac0 2024-02-19T18:52:57.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.833+0000 7ff3e9d9b700 1 --2- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff3d4036820 0x7ff3d4038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:52:57.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.833+0000 7ff3e9d9b700 1 --2- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff3d4036820 0x7ff3d4038cd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009990 tx=0x7ff3e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:52:57.856 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:57.838+0000 7ff3daffd700 1 -- 172.21.15.104:0/1223502958 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff3e4013a80 con 0x7ff3ec0fbac0 2024-02-19T18:52:58.110 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.093+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3ec04bed0 con 0x7ff3ec0fbac0 2024-02-19T18:52:58.111 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.094+0000 7ff3daffd700 1 -- 172.21.15.104:0/1223502958 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff3e4011370 con 0x7ff3ec0fbac0 2024-02-19T18:52:58.111 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:52:58.111 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:52:58.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.096+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff3d4036820 msgr2=0x7ff3d4038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:58.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.096+0000 7ff3f0856700 1 --2- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff3d4036820 0x7ff3d4038cd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff3e0009990 tx=0x7ff3e0006e40 comp rx=0 tx=0).stop 2024-02-19T18:52:58.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.096+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 msgr2=0x7ff3ec0f9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:52:58.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.096+0000 7ff3f0856700 1 --2- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 0x7ff3ec0f9210 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009680 tx=0x7ff3e4009e30 comp rx=0 tx=0).stop 2024-02-19T18:52:58.113 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.097+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 shutdown_connections 2024-02-19T18:52:58.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.097+0000 7ff3f0856700 1 --2- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff3d4036820 0x7ff3d4038cd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:58.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.097+0000 7ff3f0856700 1 --2- 172.21.15.104:0/1223502958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3ec0fbac0 0x7ff3ec0f9210 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:52:58.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.097+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 >> 172.21.15.104:0/1223502958 conn(0x7ff3ec0f71a0 msgr2=0x7ff3ec066410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:52:58.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.097+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 shutdown_connections 2024-02-19T18:52:58.114 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:52:58.097+0000 7ff3f0856700 1 -- 172.21.15.104:0/1223502958 wait complete. 2024-02-19T18:52:58.114 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:52:59.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:58 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1223502958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:52:59.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:52:58 smithi077 ceph-mon[29247]: pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:52:59.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:52:59.716 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:52:59.963 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:00.009 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:01.381 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.363+0000 7f59ee247700 1 -- 172.21.15.104:0/3629706548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 msgr2=0x7f59e80fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:01.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.363+0000 7f59ee247700 1 --2- 172.21.15.104:0/3629706548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 0x7f59e80fcfe0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f59d80048c0 tx=0x7f59d8009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:01.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.363+0000 7f59ee247700 1 -- 172.21.15.104:0/3629706548 shutdown_connections 2024-02-19T18:53:01.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.363+0000 7f59ee247700 1 --2- 172.21.15.104:0/3629706548 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 0x7f59e80fcfe0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:01.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.363+0000 7f59ee247700 1 -- 172.21.15.104:0/3629706548 >> 172.21.15.104:0/3629706548 conn(0x7f59e80f8290 msgr2=0x7f59e80fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:01.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.363+0000 7f59ee247700 1 -- 172.21.15.104:0/3629706548 shutdown_connections 2024-02-19T18:53:01.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.363+0000 7f59ee247700 1 -- 172.21.15.104:0/3629706548 wait complete. 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.363+0000 7f59ee247700 1 Processor -- start 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.364+0000 7f59ee247700 1 -- start start 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.364+0000 7f59ee247700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 0x7f59e810f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.364+0000 7f59ee247700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e810f650 con 0x7f59e80fcbd0 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.364+0000 7f59e77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 0x7f59e810f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.364+0000 7f59e77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 0x7f59e810f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:51194/0 (socket says 172.21.15.104:51194) 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.364+0000 7f59e77fe700 1 -- 172.21.15.104:0/2402837958 learned_addr learned my addr 172.21.15.104:0/2402837958 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.364+0000 7f59e77fe700 1 -- 172.21.15.104:0/2402837958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8005040 con 0x7f59e80fcbd0 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.364+0000 7f59e77fe700 1 --2- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 0x7f59e810f110 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f59d8009680 tx=0x7f59d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.365+0000 7f59e4ff9700 1 -- 172.21.15.104:0/2402837958 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59d800bed0 con 0x7f59e80fcbd0 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.365+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e810f850 con 0x7f59e80fcbd0 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.365+0000 7f59e4ff9700 1 -- 172.21.15.104:0/2402837958 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f59d800b5c0 con 0x7f59e80fcbd0 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.365+0000 7f59e4ff9700 1 -- 172.21.15.104:0/2402837958 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59d801c5b0 con 0x7f59e80fcbd0 2024-02-19T18:53:01.479 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.365+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e81104a0 con 0x7f59e80fcbd0 2024-02-19T18:53:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.365+0000 7f59e4ff9700 1 -- 172.21.15.104:0/2402837958 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f59d8013420 con 0x7f59e80fcbd0 2024-02-19T18:53:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.365+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e805f5c0 con 0x7f59e80fcbd0 2024-02-19T18:53:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.366+0000 7f59e4ff9700 1 --2- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f59d4036820 0x7f59d4038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.366+0000 7f59e4ff9700 1 -- 172.21.15.104:0/2402837958 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f59d8049b50 con 0x7f59e80fcbd0 2024-02-19T18:53:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.366+0000 7f59dffff700 1 --2- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f59d4036820 0x7f59d4038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.366+0000 7f59dffff700 1 --2- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f59d4036820 0x7f59d4038cd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009990 tx=0x7f59d0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:01.481 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.372+0000 7f59e4ff9700 1 -- 172.21.15.104:0/2402837958 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f59d8013a80 con 0x7f59e80fcbd0 2024-02-19T18:53:01.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:01 smithi077 ceph-mon[29247]: pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:01.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.624+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59e8110ff0 con 0x7f59e80fcbd0 2024-02-19T18:53:01.641 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.625+0000 7f59e4ff9700 1 -- 172.21.15.104:0/2402837958 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f59d8011370 con 0x7f59e80fcbd0 2024-02-19T18:53:01.641 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:01.642 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:01.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.626+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f59d4036820 msgr2=0x7f59d4038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:01.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.626+0000 7f59ee247700 1 --2- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f59d4036820 0x7f59d4038cd0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f59d0009990 tx=0x7f59d0006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:01.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.626+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 msgr2=0x7f59e810f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:01.643 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.626+0000 7f59ee247700 1 --2- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 0x7f59e810f110 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f59d8009680 tx=0x7f59d8009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.627+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 shutdown_connections 2024-02-19T18:53:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.627+0000 7f59ee247700 1 --2- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f59d4036820 0x7f59d4038cd0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.627+0000 7f59ee247700 1 --2- 172.21.15.104:0/2402837958 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f59e80fcbd0 0x7f59e810f110 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.627+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 >> 172.21.15.104:0/2402837958 conn(0x7f59e80f8290 msgr2=0x7f59e80f8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.627+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 shutdown_connections 2024-02-19T18:53:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:01.627+0000 7f59ee247700 1 -- 172.21.15.104:0/2402837958 wait complete. 2024-02-19T18:53:01.644 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:02.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:02 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2402837958' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:03.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:03.188 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:03.434 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:03.481 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:03.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:03 smithi077 ceph-mon[29247]: pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 -- 172.21.15.104:0/3103493557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 msgr2=0x7fbb380fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:04.989 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 --2- 172.21.15.104:0/3103493557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 0x7fbb380fbed0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbb280048c0 tx=0x7fbb28009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 -- 172.21.15.104:0/3103493557 shutdown_connections 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 --2- 172.21.15.104:0/3103493557 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 0x7fbb380fbed0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 -- 172.21.15.104:0/3103493557 >> 172.21.15.104:0/3103493557 conn(0x7fbb380f71a0 msgr2=0x7fbb380f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 -- 172.21.15.104:0/3103493557 shutdown_connections 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 -- 172.21.15.104:0/3103493557 wait complete. 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 Processor -- start 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.973+0000 7fbb3f633700 1 -- start start 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.974+0000 7fbb3f633700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 0x7fbb380f9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.974+0000 7fbb3f633700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb380f76d0 con 0x7fbb380fbac0 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.974+0000 7fbb3d3cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 0x7fbb380f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:04.990 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.974+0000 7fbb3d3cf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 0x7fbb380f9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:51220/0 (socket says 172.21.15.104:51220) 2024-02-19T18:53:04.991 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.974+0000 7fbb3d3cf700 1 -- 172.21.15.104:0/3721835882 learned_addr learned my addr 172.21.15.104:0/3721835882 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:04.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.974+0000 7fbb3d3cf700 1 -- 172.21.15.104:0/3721835882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb28005040 con 0x7fbb380fbac0 2024-02-19T18:53:04.992 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.974+0000 7fbb3d3cf700 1 --2- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 0x7fbb380f9250 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009680 tx=0x7fbb28009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.975+0000 7fbb2dffb700 1 -- 172.21.15.104:0/3721835882 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb2800bed0 con 0x7fbb380fbac0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.975+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb380f78d0 con 0x7fbb380fbac0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.975+0000 7fbb2dffb700 1 -- 172.21.15.104:0/3721835882 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbb2800b5c0 con 0x7fbb380fbac0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.975+0000 7fbb2dffb700 1 -- 172.21.15.104:0/3721835882 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb2801c5b0 con 0x7fbb380fbac0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.975+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb380f7dc0 con 0x7fbb380fbac0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.976+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb381068a0 con 0x7fbb380fbac0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.976+0000 7fbb2dffb700 1 -- 172.21.15.104:0/3721835882 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fbb28013a80 con 0x7fbb380fbac0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.976+0000 7fbb2dffb700 1 --2- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fbb24036820 0x7fbb24038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.976+0000 7fbb2dffb700 1 -- 172.21.15.104:0/3721835882 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbb28049b70 con 0x7fbb380fbac0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.976+0000 7fbb3cbce700 1 --2- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fbb24036820 0x7fbb24038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:04.993 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.977+0000 7fbb3cbce700 1 --2- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fbb24036820 0x7fbb24038cd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009990 tx=0x7fbb34006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:04.996 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:04.981+0000 7fbb2dffb700 1 -- 172.21.15.104:0/3721835882 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fbb28011960 con 0x7fbb380fbac0 2024-02-19T18:53:05.251 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.236+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb3804bed0 con 0x7fbb380fbac0 2024-02-19T18:53:05.252 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.236+0000 7fbb2dffb700 1 -- 172.21.15.104:0/3721835882 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbb28011370 con 0x7fbb380fbac0 2024-02-19T18:53:05.252 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:05.252 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fbb24036820 msgr2=0x7fbb24038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 --2- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fbb24036820 0x7fbb24038cd0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbb34009990 tx=0x7fbb34006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 msgr2=0x7fbb380f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 --2- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 0x7fbb380f9250 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009680 tx=0x7fbb28009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 shutdown_connections 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 --2- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fbb24036820 0x7fbb24038cd0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 --2- 172.21.15.104:0/3721835882 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fbb380fbac0 0x7fbb380f9250 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 >> 172.21.15.104:0/3721835882 conn(0x7fbb380f71a0 msgr2=0x7fbb380663f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 shutdown_connections 2024-02-19T18:53:05.254 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:05.238+0000 7fbb3f633700 1 -- 172.21.15.104:0/3721835882 wait complete. 2024-02-19T18:53:05.255 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:05.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:05 smithi077 ceph-mon[29247]: pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:05.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:05 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3721835882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:06.797 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:06.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:07.046 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:07.091 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:07.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:07 smithi077 ceph-mon[29247]: pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.393+0000 7fb1fda11700 1 -- 172.21.15.104:0/1314291876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 msgr2=0x7fb1f80ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.393+0000 7fb1fda11700 1 --2- 172.21.15.104:0/1314291876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 0x7fb1f80ffaf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8009c60 tx=0x7fb1e8004d10 comp rx=0 tx=0).stop 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.393+0000 7fb1fda11700 1 -- 172.21.15.104:0/1314291876 shutdown_connections 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.393+0000 7fb1fda11700 1 --2- 172.21.15.104:0/1314291876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 0x7fb1f80ffaf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.393+0000 7fb1fda11700 1 -- 172.21.15.104:0/1314291876 >> 172.21.15.104:0/1314291876 conn(0x7fb1f80f7190 msgr2=0x7fb1f80f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.393+0000 7fb1fda11700 1 -- 172.21.15.104:0/1314291876 shutdown_connections 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.393+0000 7fb1fda11700 1 -- 172.21.15.104:0/1314291876 wait complete. 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.394+0000 7fb1fda11700 1 Processor -- start 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.394+0000 7fb1fda11700 1 -- start start 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.394+0000 7fb1fda11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 0x7fb1f810cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:08.410 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.394+0000 7fb1fda11700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f810d3e0 con 0x7fb1f80fd710 2024-02-19T18:53:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.394+0000 7fb1f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 0x7fb1f810cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.394+0000 7fb1f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 0x7fb1f810cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:53190/0 (socket says 172.21.15.104:53190) 2024-02-19T18:53:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.394+0000 7fb1f6ffd700 1 -- 172.21.15.104:0/3436887540 learned_addr learned my addr 172.21.15.104:0/3436887540 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:08.411 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.394+0000 7fb1f6ffd700 1 -- 172.21.15.104:0/3436887540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e8009940 con 0x7fb1f80fd710 2024-02-19T18:53:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.395+0000 7fb1f6ffd700 1 --2- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 0x7fb1f810cea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8004e60 tx=0x7fb1e8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.395+0000 7fb1effff700 1 -- 172.21.15.104:0/3436887540 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e801d440 con 0x7fb1f80fd710 2024-02-19T18:53:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.395+0000 7fb1effff700 1 -- 172.21.15.104:0/3436887540 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb1e801da70 con 0x7fb1f80fd710 2024-02-19T18:53:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.395+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f810d5e0 con 0x7fb1f80fd710 2024-02-19T18:53:08.412 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.395+0000 7fb1effff700 1 -- 172.21.15.104:0/3436887540 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1e8012aa0 con 0x7fb1f80fd710 2024-02-19T18:53:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.395+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f810e280 con 0x7fb1f80fd710 2024-02-19T18:53:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.396+0000 7fb1effff700 1 -- 172.21.15.104:0/3436887540 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fb1e8019040 con 0x7fb1f80fd710 2024-02-19T18:53:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.396+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f805f5c0 con 0x7fb1f80fd710 2024-02-19T18:53:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.396+0000 7fb1effff700 1 --2- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb1e4036870 0x7fb1e4038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.396+0000 7fb1effff700 1 -- 172.21.15.104:0/3436887540 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb1e804a390 con 0x7fb1f80fd710 2024-02-19T18:53:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.396+0000 7fb1ee7fc700 1 --2- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb1e4036870 0x7fb1e4038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:08.413 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.397+0000 7fb1ee7fc700 1 --2- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb1e4036870 0x7fb1e4038d20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0009990 tx=0x7fb1e0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:08.416 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.401+0000 7fb1effff700 1 -- 172.21.15.104:0/3436887540 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb1e8012410 con 0x7fb1f80fd710 2024-02-19T18:53:08.670 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.654+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1f810ed70 con 0x7fb1f80fd710 2024-02-19T18:53:08.671 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.655+0000 7fb1effff700 1 -- 172.21.15.104:0/3436887540 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb1e8017020 con 0x7fb1f80fd710 2024-02-19T18:53:08.671 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:08.671 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb1e4036870 msgr2=0x7fb1e4038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 --2- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb1e4036870 0x7fb1e4038d20 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb1e0009990 tx=0x7fb1e0006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 msgr2=0x7fb1f810cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 --2- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 0x7fb1f810cea0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb1e8004e60 tx=0x7fb1e8004d10 comp rx=0 tx=0).stop 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 shutdown_connections 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 --2- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fb1e4036870 0x7fb1e4038d20 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 --2- 172.21.15.104:0/3436887540 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb1f80fd710 0x7fb1f810cea0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 >> 172.21.15.104:0/3436887540 conn(0x7fb1f80f7190 msgr2=0x7fb1f80f7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 shutdown_connections 2024-02-19T18:53:08.673 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:08.657+0000 7fb1fda11700 1 -- 172.21.15.104:0/3436887540 wait complete. 2024-02-19T18:53:08.674 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:09.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:08 smithi077 ceph-mon[29247]: pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:09.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:08 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3436887540' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:10.156 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:10.157 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:10.402 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:10.451 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:11.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:11 smithi077 ceph-mon[29247]: pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:11.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.807+0000 7f209a322700 1 -- 172.21.15.104:0/3319027961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 msgr2=0x7f20940fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:11.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.807+0000 7f209a322700 1 --2- 172.21.15.104:0/3319027961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 0x7f20940fc340 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f20840048c0 tx=0x7f2084009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:11.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.807+0000 7f209a322700 1 -- 172.21.15.104:0/3319027961 shutdown_connections 2024-02-19T18:53:11.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.807+0000 7f209a322700 1 --2- 172.21.15.104:0/3319027961 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 0x7f20940fc340 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:11.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.807+0000 7f209a322700 1 -- 172.21.15.104:0/3319027961 >> 172.21.15.104:0/3319027961 conn(0x7f20940f69c0 msgr2=0x7f20940f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:11.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.807+0000 7f209a322700 1 -- 172.21.15.104:0/3319027961 shutdown_connections 2024-02-19T18:53:11.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.807+0000 7f209a322700 1 -- 172.21.15.104:0/3319027961 wait complete. 2024-02-19T18:53:11.823 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.808+0000 7f209a322700 1 Processor -- start 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.808+0000 7f209a322700 1 -- start start 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.808+0000 7f209a322700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 0x7f2094113550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.808+0000 7f209a322700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2094113a90 con 0x7f20940fbf30 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.808+0000 7f2093fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 0x7f2094113550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.808+0000 7f2093fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 0x7f2094113550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:53206/0 (socket says 172.21.15.104:53206) 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.808+0000 7f2093fff700 1 -- 172.21.15.104:0/3915523888 learned_addr learned my addr 172.21.15.104:0/3915523888 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.808+0000 7f2093fff700 1 -- 172.21.15.104:0/3915523888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2084005040 con 0x7f20940fbf30 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.809+0000 7f2093fff700 1 --2- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 0x7f2094113550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2084009300 tx=0x7f2084009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.809+0000 7f2090ff9700 1 -- 172.21.15.104:0/3915523888 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2084004e60 con 0x7f20940fbf30 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.809+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2094113c90 con 0x7f20940fbf30 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.809+0000 7f2090ff9700 1 -- 172.21.15.104:0/3915523888 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2084009680 con 0x7f20940fbf30 2024-02-19T18:53:11.825 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.809+0000 7f2090ff9700 1 -- 172.21.15.104:0/3915523888 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f208401c5b0 con 0x7f20940fbf30 2024-02-19T18:53:11.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.809+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2094114930 con 0x7f20940fbf30 2024-02-19T18:53:11.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.810+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209405f5c0 con 0x7f20940fbf30 2024-02-19T18:53:11.826 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.810+0000 7f2090ff9700 1 -- 172.21.15.104:0/3915523888 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f20840097f0 con 0x7f20940fbf30 2024-02-19T18:53:11.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.810+0000 7f2090ff9700 1 --2- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f207c036870 0x7f207c038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:11.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.810+0000 7f2090ff9700 1 -- 172.21.15.104:0/3915523888 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2084049a40 con 0x7f20940fbf30 2024-02-19T18:53:11.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.810+0000 7f20937fe700 1 --2- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f207c036870 0x7f207c038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:11.827 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.811+0000 7f20937fe700 1 --2- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f207c036870 0x7f207c038d20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2088009990 tx=0x7f2088006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:11.830 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:11.815+0000 7f2090ff9700 1 -- 172.21.15.104:0/3915523888 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2084011550 con 0x7f20940fbf30 2024-02-19T18:53:12.085 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.070+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2094115480 con 0x7f20940fbf30 2024-02-19T18:53:12.086 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.070+0000 7f2090ff9700 1 -- 172.21.15.104:0/3915523888 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f2084011370 con 0x7f20940fbf30 2024-02-19T18:53:12.086 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:12.086 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.072+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f207c036870 msgr2=0x7f207c038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.072+0000 7f209a322700 1 --2- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f207c036870 0x7f207c038d20 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2088009990 tx=0x7f2088006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.072+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 msgr2=0x7f2094113550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.072+0000 7f209a322700 1 --2- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 0x7f2094113550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f2084009300 tx=0x7f2084009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.072+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 shutdown_connections 2024-02-19T18:53:12.088 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.072+0000 7f209a322700 1 --2- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f207c036870 0x7f207c038d20 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.072+0000 7f209a322700 1 --2- 172.21.15.104:0/3915523888 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f20940fbf30 0x7f2094113550 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.072+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 >> 172.21.15.104:0/3915523888 conn(0x7f20940f69c0 msgr2=0x7f20940fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.073+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 shutdown_connections 2024-02-19T18:53:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:12.073+0000 7f209a322700 1 -- 172.21.15.104:0/3915523888 wait complete. 2024-02-19T18:53:12.089 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:12.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:12 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3915523888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:13.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:13 smithi077 ceph-mon[29247]: pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:13.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:13.650 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:13.897 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:13.945 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:15.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.198+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3371927467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 msgr2=0x7fd8180fbee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:15.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.198+0000 7fd81e7fc700 1 --2- 172.21.15.104:0/3371927467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 0x7fd8180fbee0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd8000048c0 tx=0x7fd800009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:15.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.198+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3371927467 shutdown_connections 2024-02-19T18:53:15.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.198+0000 7fd81e7fc700 1 --2- 172.21.15.104:0/3371927467 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 0x7fd8180fbee0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:15.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.198+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3371927467 >> 172.21.15.104:0/3371927467 conn(0x7fd8180f7190 msgr2=0x7fd8180f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:15.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.198+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3371927467 shutdown_connections 2024-02-19T18:53:15.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.198+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3371927467 wait complete. 2024-02-19T18:53:15.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.199+0000 7fd81e7fc700 1 Processor -- start 2024-02-19T18:53:15.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.199+0000 7fd81e7fc700 1 -- start start 2024-02-19T18:53:15.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.199+0000 7fd81e7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 0x7fd8180f9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:15.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.199+0000 7fd81e7fc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8180f76d0 con 0x7fd8180fbad0 2024-02-19T18:53:15.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.199+0000 7fd817fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 0x7fd8180f9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:15.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd817fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 0x7fd8180f9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:53224/0 (socket says 172.21.15.104:53224) 2024-02-19T18:53:15.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd817fff700 1 -- 172.21.15.104:0/3933288144 learned_addr learned my addr 172.21.15.104:0/3933288144 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:15.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd817fff700 1 -- 172.21.15.104:0/3933288144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800005040 con 0x7fd8180fbad0 2024-02-19T18:53:15.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd817fff700 1 --2- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 0x7fd8180f9250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd800009680 tx=0x7fd800009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:15.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd814ff9700 1 -- 172.21.15.104:0/3933288144 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd80000bed0 con 0x7fd8180fbad0 2024-02-19T18:53:15.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8180f78d0 con 0x7fd8180fbad0 2024-02-19T18:53:15.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd814ff9700 1 -- 172.21.15.104:0/3933288144 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd80000b5c0 con 0x7fd8180fbad0 2024-02-19T18:53:15.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd814ff9700 1 -- 172.21.15.104:0/3933288144 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd800025560 con 0x7fd8180fbad0 2024-02-19T18:53:15.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.200+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8180f7dc0 con 0x7fd8180fbad0 2024-02-19T18:53:15.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.201+0000 7fd814ff9700 1 -- 172.21.15.104:0/3933288144 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fd800013420 con 0x7fd8180fbad0 2024-02-19T18:53:15.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.201+0000 7fd814ff9700 1 --2- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd804036820 0x7fd804038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:15.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.201+0000 7fd814ff9700 1 -- 172.21.15.104:0/3933288144 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd800049b40 con 0x7fd8180fbad0 2024-02-19T18:53:15.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.201+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8181068b0 con 0x7fd8180fbad0 2024-02-19T18:53:15.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.201+0000 7fd8177fe700 1 --2- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd804036820 0x7fd804038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:15.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.202+0000 7fd8177fe700 1 --2- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd804036820 0x7fd804038cd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd808009990 tx=0x7fd808006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:15.222 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.206+0000 7fd814ff9700 1 -- 172.21.15.104:0/3933288144 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd800011550 con 0x7fd8180fbad0 2024-02-19T18:53:15.477 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.462+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd81804bed0 con 0x7fd8180fbad0 2024-02-19T18:53:15.478 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.462+0000 7fd814ff9700 1 -- 172.21.15.104:0/3933288144 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd800011370 con 0x7fd8180fbad0 2024-02-19T18:53:15.478 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:15.478 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd804036820 msgr2=0x7fd804038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 --2- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd804036820 0x7fd804038cd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd808009990 tx=0x7fd808006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 msgr2=0x7fd8180f9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 --2- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 0x7fd8180f9250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd800009680 tx=0x7fd800009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 shutdown_connections 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 --2- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd804036820 0x7fd804038cd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 --2- 172.21.15.104:0/3933288144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8180fbad0 0x7fd8180f9250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 >> 172.21.15.104:0/3933288144 conn(0x7fd8180f7190 msgr2=0x7fd818065960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 shutdown_connections 2024-02-19T18:53:15.480 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:15.464+0000 7fd81e7fc700 1 -- 172.21.15.104:0/3933288144 wait complete. 2024-02-19T18:53:15.481 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:15.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:15 smithi077 ceph-mon[29247]: pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:16.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:16 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3933288144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:17.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:17.096 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:17.363 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:17.410 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:17.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:17 smithi077 ceph-mon[29247]: pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:18.876 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 -- 172.21.15.104:0/1017133608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 msgr2=0x7f598c0fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:18.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 --2- 172.21.15.104:0/1017133608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 0x7f598c0fbed0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f597c0048c0 tx=0x7f597c009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:18.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 -- 172.21.15.104:0/1017133608 shutdown_connections 2024-02-19T18:53:18.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 --2- 172.21.15.104:0/1017133608 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 0x7f598c0fbed0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:18.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 -- 172.21.15.104:0/1017133608 >> 172.21.15.104:0/1017133608 conn(0x7f598c0f71a0 msgr2=0x7f598c0f95b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:18.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 -- 172.21.15.104:0/1017133608 shutdown_connections 2024-02-19T18:53:18.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 -- 172.21.15.104:0/1017133608 wait complete. 2024-02-19T18:53:18.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 Processor -- start 2024-02-19T18:53:18.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.861+0000 7f5992683700 1 -- start start 2024-02-19T18:53:18.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.862+0000 7f5992683700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 0x7f598c0f9210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:18.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.862+0000 7f5992683700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c0f7690 con 0x7f598c0fbac0 2024-02-19T18:53:18.878 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.862+0000 7f598bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 0x7f598c0f9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:18.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.862+0000 7f598bfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 0x7f598c0f9210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:40804/0 (socket says 172.21.15.104:40804) 2024-02-19T18:53:18.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.862+0000 7f598bfff700 1 -- 172.21.15.104:0/117262691 learned_addr learned my addr 172.21.15.104:0/117262691 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:18.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.862+0000 7f598bfff700 1 -- 172.21.15.104:0/117262691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597c005040 con 0x7f598c0fbac0 2024-02-19T18:53:18.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.862+0000 7f598bfff700 1 --2- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 0x7f598c0f9210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f597c009b80 tx=0x7f597c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:18.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.863+0000 7f5988ff9700 1 -- 172.21.15.104:0/117262691 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f597c00b5c0 con 0x7f598c0fbac0 2024-02-19T18:53:18.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.863+0000 7f5988ff9700 1 -- 172.21.15.104:0/117262691 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f597c00be70 con 0x7f598c0fbac0 2024-02-19T18:53:18.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.863+0000 7f5988ff9700 1 -- 172.21.15.104:0/117262691 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f597c01c660 con 0x7f598c0fbac0 2024-02-19T18:53:18.879 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.863+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c0f7890 con 0x7f598c0fbac0 2024-02-19T18:53:18.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.863+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c0f7cc0 con 0x7f598c0fbac0 2024-02-19T18:53:18.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.863+0000 7f5988ff9700 1 -- 172.21.15.104:0/117262691 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7f597c013420 con 0x7f598c0fbac0 2024-02-19T18:53:18.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.863+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c1068a0 con 0x7f598c0fbac0 2024-02-19T18:53:18.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.864+0000 7f5988ff9700 1 --2- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5978036870 0x7f5978038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:18.880 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.864+0000 7f5988ff9700 1 -- 172.21.15.104:0/117262691 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f597c049a80 con 0x7f598c0fbac0 2024-02-19T18:53:18.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.864+0000 7f598b7fe700 1 --2- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5978036870 0x7f5978038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:18.881 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.864+0000 7f598b7fe700 1 --2- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5978036870 0x7f5978038d20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5974009990 tx=0x7f5974006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:18.884 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:18.869+0000 7f5988ff9700 1 -- 172.21.15.104:0/117262691 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f597c011550 con 0x7f598c0fbac0 2024-02-19T18:53:19.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.121+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f598c04bed0 con 0x7f598c0fbac0 2024-02-19T18:53:19.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.121+0000 7f5988ff9700 1 -- 172.21.15.104:0/117262691 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f597c011370 con 0x7f598c0fbac0 2024-02-19T18:53:19.137 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:19.137 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5978036870 msgr2=0x7f5978038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 --2- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5978036870 0x7f5978038d20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5974009990 tx=0x7f5974006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 msgr2=0x7f598c0f9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 --2- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 0x7f598c0f9210 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f597c009b80 tx=0x7f597c009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 shutdown_connections 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 --2- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f5978036870 0x7f5978038d20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 --2- 172.21.15.104:0/117262691 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f598c0fbac0 0x7f598c0f9210 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 >> 172.21.15.104:0/117262691 conn(0x7f598c0f71a0 msgr2=0x7f598c066410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:19.139 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 shutdown_connections 2024-02-19T18:53:19.140 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:19.123+0000 7f5992683700 1 -- 172.21.15.104:0/117262691 wait complete. 2024-02-19T18:53:19.140 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:19.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:18 smithi077 ceph-mon[29247]: pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:20.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:19 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/117262691' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:20.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:20.675 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:20.923 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:20.970 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:21.160 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:20 smithi077 ceph-mon[29247]: pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.422+0000 7fe591935700 1 -- 172.21.15.104:0/1888814730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 msgr2=0x7fe58c0ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.422+0000 7fe591935700 1 --2- 172.21.15.104:0/1888814730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 0x7fe58c0ffaf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0048c0 tx=0x7fe57c009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.422+0000 7fe591935700 1 -- 172.21.15.104:0/1888814730 shutdown_connections 2024-02-19T18:53:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.422+0000 7fe591935700 1 --2- 172.21.15.104:0/1888814730 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 0x7fe58c0ffaf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.422+0000 7fe591935700 1 -- 172.21.15.104:0/1888814730 >> 172.21.15.104:0/1888814730 conn(0x7fe58c0f7190 msgr2=0x7fe58c0f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.422+0000 7fe591935700 1 -- 172.21.15.104:0/1888814730 shutdown_connections 2024-02-19T18:53:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.422+0000 7fe591935700 1 -- 172.21.15.104:0/1888814730 wait complete. 2024-02-19T18:53:22.439 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.423+0000 7fe591935700 1 Processor -- start 2024-02-19T18:53:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.423+0000 7fe591935700 1 -- start start 2024-02-19T18:53:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.423+0000 7fe591935700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 0x7fe58c0fd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.423+0000 7fe591935700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58c0fb620 con 0x7fe58c0fd710 2024-02-19T18:53:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.423+0000 7fe58affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 0x7fe58c0fd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.423+0000 7fe58affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 0x7fe58c0fd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:40828/0 (socket says 172.21.15.104:40828) 2024-02-19T18:53:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.423+0000 7fe58affd700 1 -- 172.21.15.104:0/2827626406 learned_addr learned my addr 172.21.15.104:0/2827626406 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:22.440 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.423+0000 7fe58affd700 1 -- 172.21.15.104:0/2827626406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe57c005040 con 0x7fe58c0fd710 2024-02-19T18:53:22.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.424+0000 7fe58affd700 1 --2- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 0x7fe58c0fd1a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009b80 tx=0x7fe57c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:22.441 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.424+0000 7fe5737fe700 1 -- 172.21.15.104:0/2827626406 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe57c00b5c0 con 0x7fe58c0fd710 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.424+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe58c0fb820 con 0x7fe58c0fd710 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.424+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe58c0fbd10 con 0x7fe58c0fd710 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.424+0000 7fe5737fe700 1 -- 172.21.15.104:0/2827626406 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe57c00be70 con 0x7fe58c0fd710 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.424+0000 7fe5737fe700 1 -- 172.21.15.104:0/2827626406 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe57c0255c0 con 0x7fe58c0fd710 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.425+0000 7fe5737fe700 1 -- 172.21.15.104:0/2827626406 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fe57c01c3f0 con 0x7fe58c0fd710 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.425+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5780052e0 con 0x7fe58c0fd710 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.425+0000 7fe5737fe700 1 --2- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe574036820 0x7fe574038cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.425+0000 7fe5737fe700 1 -- 172.21.15.104:0/2827626406 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe57c049b00 con 0x7fe58c0fd710 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.425+0000 7fe58a7fc700 1 --2- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe574036820 0x7fe574038cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:22.442 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.426+0000 7fe58a7fc700 1 --2- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe574036820 0x7fe574038cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe580009990 tx=0x7fe580006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:22.446 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.430+0000 7fe5737fe700 1 -- 172.21.15.104:0/2827626406 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe57c01cb60 con 0x7fe58c0fd710 2024-02-19T18:53:22.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.684+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe578005590 con 0x7fe58c0fd710 2024-02-19T18:53:22.700 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.685+0000 7fe5737fe700 1 -- 172.21.15.104:0/2827626406 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe57c01a2f0 con 0x7fe58c0fd710 2024-02-19T18:53:22.701 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:22.701 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:22.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe574036820 msgr2=0x7fe574038cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:22.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 --2- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe574036820 0x7fe574038cd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe580009990 tx=0x7fe580006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:22.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 msgr2=0x7fe58c0fd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:22.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 --2- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 0x7fe58c0fd1a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe57c009b80 tx=0x7fe57c009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:22.703 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 shutdown_connections 2024-02-19T18:53:22.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 --2- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fe574036820 0x7fe574038cd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:22.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 --2- 172.21.15.104:0/2827626406 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe58c0fd710 0x7fe58c0fd1a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:22.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 >> 172.21.15.104:0/2827626406 conn(0x7fe58c0f7190 msgr2=0x7fe58c0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:22.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 shutdown_connections 2024-02-19T18:53:22.704 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:22.687+0000 7fe591935700 1 -- 172.21.15.104:0/2827626406 wait complete. 2024-02-19T18:53:22.704 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:23.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:23 smithi077 ceph-mon[29247]: pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:23.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:23 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2827626406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:24.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:24.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:24.525 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:24.571 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:25.532 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:25 smithi077 ceph-mon[29247]: pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 -- 172.21.15.104:0/4124830336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 msgr2=0x7fd6980fe610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 --2- 172.21.15.104:0/4124830336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 0x7fd6980fe610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd6880048c0 tx=0x7fd688009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 -- 172.21.15.104:0/4124830336 shutdown_connections 2024-02-19T18:53:25.950 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 --2- 172.21.15.104:0/4124830336 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 0x7fd6980fe610 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 -- 172.21.15.104:0/4124830336 >> 172.21.15.104:0/4124830336 conn(0x7fd6980f69c0 msgr2=0x7fd6980f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 -- 172.21.15.104:0/4124830336 shutdown_connections 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 -- 172.21.15.104:0/4124830336 wait complete. 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 Processor -- start 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.933+0000 7fd69dd29700 1 -- start start 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd69dd29700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 0x7fd6980fda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd69dd29700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6980fbec0 con 0x7fd6980fe200 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd6977fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 0x7fd6980fda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd6977fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 0x7fd6980fda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:40844/0 (socket says 172.21.15.104:40844) 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd6977fe700 1 -- 172.21.15.104:0/3648676371 learned_addr learned my addr 172.21.15.104:0/3648676371 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd6977fe700 1 -- 172.21.15.104:0/3648676371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd688005040 con 0x7fd6980fe200 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd6977fe700 1 --2- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 0x7fd6980fda40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd688009300 tx=0x7fd688009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd67ffff700 1 -- 172.21.15.104:0/3648676371 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd688004e60 con 0x7fd6980fe200 2024-02-19T18:53:25.951 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd67ffff700 1 -- 172.21.15.104:0/3648676371 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd688009680 con 0x7fd6980fe200 2024-02-19T18:53:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6980fc0c0 con 0x7fd6980fe200 2024-02-19T18:53:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.934+0000 7fd67ffff700 1 -- 172.21.15.104:0/3648676371 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd68801c5b0 con 0x7fd6980fe200 2024-02-19T18:53:25.952 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.935+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6980fc5b0 con 0x7fd6980fe200 2024-02-19T18:53:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.935+0000 7fd67ffff700 1 -- 172.21.15.104:0/3648676371 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 13) v1 ==== 43424+0+0 (secure 0 0 0) 0x7fd6880097f0 con 0x7fd6980fe200 2024-02-19T18:53:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.935+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd698106910 con 0x7fd6980fe200 2024-02-19T18:53:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.935+0000 7fd67ffff700 1 --2- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd680036870 0x7fd680038d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.936+0000 7fd67ffff700 1 -- 172.21.15.104:0/3648676371 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd688049a40 con 0x7fd6980fe200 2024-02-19T18:53:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.936+0000 7fd696ffd700 1 --2- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd680036870 0x7fd680038d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:25.953 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.936+0000 7fd696ffd700 1 --2- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd680036870 0x7fd680038d20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009990 tx=0x7fd68c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:25.956 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:25.940+0000 7fd67ffff700 1 -- 172.21.15.104:0/3648676371 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd688011550 con 0x7fd6980fe200 2024-02-19T18:53:26.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.198+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd69804bed0 con 0x7fd6980fe200 2024-02-19T18:53:26.214 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.199+0000 7fd67ffff700 1 -- 172.21.15.104:0/3648676371 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fd688011370 con 0x7fd6980fe200 2024-02-19T18:53:26.215 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:26.215 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:26.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd680036870 msgr2=0x7fd680038d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:26.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 --2- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd680036870 0x7fd680038d20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd68c009990 tx=0x7fd68c006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:26.216 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 msgr2=0x7fd6980fda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 --2- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 0x7fd6980fda40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd688009300 tx=0x7fd688009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 shutdown_connections 2024-02-19T18:53:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 --2- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7fd680036870 0x7fd680038d20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 --2- 172.21.15.104:0/3648676371 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6980fe200 0x7fd6980fda40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 >> 172.21.15.104:0/3648676371 conn(0x7fd6980f69c0 msgr2=0x7fd6980f7760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 shutdown_connections 2024-02-19T18:53:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:26.201+0000 7fd69dd29700 1 -- 172.21.15.104:0/3648676371 wait complete. 2024-02-19T18:53:26.217 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:26.443 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:26 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3648676371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:27.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:27.714 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:27.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:27 smithi077 ceph-mon[29247]: pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:27.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:27 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:27.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:27 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:27.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:27 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:27.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:27 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-19T18:53:27.964 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:28.010 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:28.864 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:28 smithi077 ceph-mon[29247]: from='mgr.14162 172.21.15.77:0/3719165407' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-19T18:53:28.864 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:28 smithi077 ceph-mon[29247]: mgrmap e14: smithi077.yiiqpd(active, since 2m) 2024-02-19T18:53:29.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.344+0000 7ff8a4dca700 1 -- 172.21.15.104:0/2668436436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 msgr2=0x7ff8a00fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:29.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.344+0000 7ff8a4dca700 1 --2- 172.21.15.104:0/2668436436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 0x7ff8a00fcfe0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff8900048c0 tx=0x7ff890009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.344+0000 7ff8a4dca700 1 -- 172.21.15.104:0/2668436436 shutdown_connections 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.344+0000 7ff8a4dca700 1 --2- 172.21.15.104:0/2668436436 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 0x7ff8a00fcfe0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.344+0000 7ff8a4dca700 1 -- 172.21.15.104:0/2668436436 >> 172.21.15.104:0/2668436436 conn(0x7ff8a00f8270 msgr2=0x7ff8a00fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.344+0000 7ff8a4dca700 1 -- 172.21.15.104:0/2668436436 shutdown_connections 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.344+0000 7ff8a4dca700 1 -- 172.21.15.104:0/2668436436 wait complete. 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff8a4dca700 1 Processor -- start 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff8a4dca700 1 -- start start 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff8a4dca700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 0x7ff8a010cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff8a4dca700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a010d4a0 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff89e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 0x7ff8a010cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff89e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 0x7ff8a010cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:59474/0 (socket says 172.21.15.104:59474) 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff89e59c700 1 -- 172.21.15.104:0/1961688301 learned_addr learned my addr 172.21.15.104:0/1961688301 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff89e59c700 1 -- 172.21.15.104:0/1961688301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890005040 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.345+0000 7ff89e59c700 1 --2- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 0x7ff8a010cf60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff890009b80 tx=0x7ff890009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:29.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.346+0000 7ff8977fe700 1 -- 172.21.15.104:0/1961688301 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff89000b5c0 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.346+0000 7ff8977fe700 1 -- 172.21.15.104:0/1961688301 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff89000be70 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.346+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a010d6a0 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.346+0000 7ff8977fe700 1 -- 172.21.15.104:0/1961688301 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff89001c660 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.346+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a010e2f0 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.346+0000 7ff8977fe700 1 -- 172.21.15.104:0/1961688301 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7ff89001c7c0 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.346+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a005f5c0 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.347+0000 7ff8977fe700 1 --2- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff88c036910 0x7ff88c038dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:29.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.347+0000 7ff8977fe700 1 -- 172.21.15.104:0/1961688301 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7ff890049ad0 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.347+0000 7ff895d9b700 1 -- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff88c036910 msgr2=0x7ff88c038dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3287530536 2024-02-19T18:53:29.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.347+0000 7ff895d9b700 1 --2- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff88c036910 0x7ff88c038dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:53:29.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.351+0000 7ff8977fe700 1 -- 172.21.15.104:0/1961688301 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff8900137a0 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.547+0000 7ff895d9b700 1 -- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff88c036910 msgr2=0x7ff88c038dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3287530536 2024-02-19T18:53:29.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.547+0000 7ff895d9b700 1 --2- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff88c036910 0x7ff88c038dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:53:29.627 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.610+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8a010ee40 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.628 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.611+0000 7ff8977fe700 1 -- 172.21.15.104:0/1961688301 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7ff890011370 con 0x7ff8a00fcbd0 2024-02-19T18:53:29.628 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:29.628 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff88c036910 msgr2=0x7ff88c038dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 --2- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff88c036910 0x7ff88c038dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 msgr2=0x7ff8a010cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 --2- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 0x7ff8a010cf60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff890009b80 tx=0x7ff890009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 shutdown_connections 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 --2- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7ff88c036910 0x7ff88c038dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 --2- 172.21.15.104:0/1961688301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8a00fcbd0 0x7ff8a010cf60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 >> 172.21.15.104:0/1961688301 conn(0x7ff8a00f8270 msgr2=0x7ff8a00f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 shutdown_connections 2024-02-19T18:53:29.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:29.613+0000 7ff8a4dca700 1 -- 172.21.15.104:0/1961688301 wait complete. 2024-02-19T18:53:29.631 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:29.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:29 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1961688301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:31.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:31.201 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:31.446 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:31.493 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.853+0000 7f12d05a4700 1 -- 172.21.15.104:0/2615425630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 msgr2=0x7f12c80ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.853+0000 7f12d05a4700 1 --2- 172.21.15.104:0/2615425630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 0x7f12c80ffaf0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f12c0009c60 tx=0x7f12c0004d10 comp rx=0 tx=0).stop 2024-02-19T18:53:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.853+0000 7f12d05a4700 1 -- 172.21.15.104:0/2615425630 shutdown_connections 2024-02-19T18:53:32.871 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.853+0000 7f12d05a4700 1 --2- 172.21.15.104:0/2615425630 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 0x7f12c80ffaf0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.853+0000 7f12d05a4700 1 -- 172.21.15.104:0/2615425630 >> 172.21.15.104:0/2615425630 conn(0x7f12c80f7190 msgr2=0x7f12c80f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.853+0000 7f12d05a4700 1 -- 172.21.15.104:0/2615425630 shutdown_connections 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.853+0000 7f12d05a4700 1 -- 172.21.15.104:0/2615425630 wait complete. 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12d05a4700 1 Processor -- start 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12d05a4700 1 -- start start 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12d05a4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 0x7f12c810f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12d05a4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c810f5e0 con 0x7f12c80fd710 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12ce340700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 0x7f12c810f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12ce340700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 0x7f12c810f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:59480/0 (socket says 172.21.15.104:59480) 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12ce340700 1 -- 172.21.15.104:0/3267695337 learned_addr learned my addr 172.21.15.104:0/3267695337 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12ce340700 1 -- 172.21.15.104:0/3267695337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c0009940 con 0x7f12c80fd710 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12ce340700 1 --2- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 0x7f12c810f0a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f12c0004e60 tx=0x7f12c0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12beffd700 1 -- 172.21.15.104:0/3267695337 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c001d440 con 0x7f12c80fd710 2024-02-19T18:53:32.872 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.854+0000 7f12beffd700 1 -- 172.21.15.104:0/3267695337 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12c001da70 con 0x7f12c80fd710 2024-02-19T18:53:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.855+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c810f7e0 con 0x7f12c80fd710 2024-02-19T18:53:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.855+0000 7f12beffd700 1 -- 172.21.15.104:0/3267695337 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12c0012aa0 con 0x7f12c80fd710 2024-02-19T18:53:32.873 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.855+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c8110480 con 0x7f12c80fd710 2024-02-19T18:53:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.855+0000 7f12beffd700 1 -- 172.21.15.104:0/3267695337 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 14) v1 ==== 43438+0+0 (secure 0 0 0) 0x7f12c0019040 con 0x7f12c80fd710 2024-02-19T18:53:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.856+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c805f5c0 con 0x7f12c80fd710 2024-02-19T18:53:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.856+0000 7f12beffd700 1 --2- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f12b40368c0 0x7f12b4038d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.856+0000 7f12beffd700 1 -- 172.21.15.104:0/3267695337 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f12c004a540 con 0x7f12c80fd710 2024-02-19T18:53:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.856+0000 7f12cdb3f700 1 -- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f12b40368c0 msgr2=0x7f12b4038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3287530536 2024-02-19T18:53:32.874 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.856+0000 7f12cdb3f700 1 --2- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f12b40368c0 0x7f12b4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:53:32.877 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:32.860+0000 7f12beffd700 1 -- 172.21.15.104:0/3267695337 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f12c002e530 con 0x7f12c80fd710 2024-02-19T18:53:33.073 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.057+0000 7f12cdb3f700 1 -- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f12b40368c0 msgr2=0x7f12b4038d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.77:6800/3287530536 2024-02-19T18:53:33.074 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.057+0000 7f12cdb3f700 1 --2- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f12b40368c0 0x7f12b4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-19T18:53:33.133 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.117+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12c8110f70 con 0x7f12c80fd710 2024-02-19T18:53:33.134 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.117+0000 7f12beffd700 1 -- 172.21.15.104:0/3267695337 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f12c0017070 con 0x7f12c80fd710 2024-02-19T18:53:33.134 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:33.134 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:33.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f12b40368c0 msgr2=0x7f12b4038d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:53:33.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 --2- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f12b40368c0 0x7f12b4038d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:33.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 msgr2=0x7f12c810f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:33.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 --2- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 0x7f12c810f0a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f12c0004e60 tx=0x7f12c0004d10 comp rx=0 tx=0).stop 2024-02-19T18:53:33.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 shutdown_connections 2024-02-19T18:53:33.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 --2- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:6800/3287530536,v1:172.21.15.77:6801/3287530536] conn(0x7f12b40368c0 0x7f12b4038d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:33.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 --2- 172.21.15.104:0/3267695337 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f12c80fd710 0x7f12c810f0a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:33.136 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 >> 172.21.15.104:0/3267695337 conn(0x7f12c80f7190 msgr2=0x7f12c80f95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 shutdown_connections 2024-02-19T18:53:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:33.119+0000 7f12d05a4700 1 -- 172.21.15.104:0/3267695337 wait complete. 2024-02-19T18:53:33.137 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:33.898 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:33 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3267695337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:34.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:34.764 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:35.017 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:35.066 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: Active manager daemon smithi077.yiiqpd restarted 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: Activating manager daemon smithi077.yiiqpd 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: osdmap e5: 0 total, 0 up, 0 in 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: mgrmap e15: smithi077.yiiqpd(active, starting, since 0.00314264s) 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi077.yiiqpd", "id": "smithi077.yiiqpd"}]: dispatch 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T18:53:35.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: Manager daemon smithi077.yiiqpd is now available 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/mirror_snapshot_schedule"}]: dispatch 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/trash_purge_schedule"}]: dispatch 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: [19/Feb/2024:18:53:34] ENGINE Bus STARTING 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:35.078 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:36.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:35 smithi077 ceph-mon[29247]: [19/Feb/2024:18:53:34] ENGINE Serving on https://172.21.15.77:7150 2024-02-19T18:53:36.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:35 smithi077 ceph-mon[29247]: [19/Feb/2024:18:53:34] ENGINE Bus STARTED 2024-02-19T18:53:36.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:35 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:36.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:35 smithi077 ceph-mon[29247]: mgrmap e16: smithi077.yiiqpd(active, since 1.00511s) 2024-02-19T18:53:36.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:35 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:36.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 -- 172.21.15.104:0/1474105881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 msgr2=0x7f0b700fcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:36.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 --2- 172.21.15.104:0/1474105881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 0x7f0b700fcfe0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0b640048c0 tx=0x7f0b64009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:36.586 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 -- 172.21.15.104:0/1474105881 shutdown_connections 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 --2- 172.21.15.104:0/1474105881 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 0x7f0b700fcfe0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 -- 172.21.15.104:0/1474105881 >> 172.21.15.104:0/1474105881 conn(0x7f0b700f8290 msgr2=0x7f0b700fa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 -- 172.21.15.104:0/1474105881 shutdown_connections 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 -- 172.21.15.104:0/1474105881 wait complete. 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 Processor -- start 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 -- start start 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 0x7f0b7010ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.568+0000 7f0b74ed1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7010d410 con 0x7f0b700fcbd0 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b6e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 0x7f0b7010ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b6e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 0x7f0b7010ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:47276/0 (socket says 172.21.15.104:47276) 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b6e59c700 1 -- 172.21.15.104:0/3546328017 learned_addr learned my addr 172.21.15.104:0/3546328017 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:36.587 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b6e59c700 1 -- 172.21.15.104:0/3546328017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64005040 con 0x7f0b700fcbd0 2024-02-19T18:53:36.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b6e59c700 1 --2- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 0x7f0b7010ced0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009680 tx=0x7f0b64009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:36.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b5effd700 1 -- 172.21.15.104:0/3546328017 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6400bed0 con 0x7f0b700fcbd0 2024-02-19T18:53:36.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7010d610 con 0x7f0b700fcbd0 2024-02-19T18:53:36.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b5effd700 1 -- 172.21.15.104:0/3546328017 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0b6400b5c0 con 0x7f0b700fcbd0 2024-02-19T18:53:36.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b5effd700 1 -- 172.21.15.104:0/3546328017 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6401c5b0 con 0x7f0b700fcbd0 2024-02-19T18:53:36.588 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.569+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7010e2b0 con 0x7f0b700fcbd0 2024-02-19T18:53:36.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.570+0000 7f0b5effd700 1 -- 172.21.15.104:0/3546328017 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f0b6401c820 con 0x7f0b700fcbd0 2024-02-19T18:53:36.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.570+0000 7f0b5effd700 1 --2- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0b580366b0 0x7f0b58038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:36.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.570+0000 7f0b5effd700 1 -- 172.21.15.104:0/3546328017 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0b64049960 con 0x7f0b700fcbd0 2024-02-19T18:53:36.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.570+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7005f5c0 con 0x7f0b700fcbd0 2024-02-19T18:53:36.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.571+0000 7f0b6dd9b700 1 --2- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0b580366b0 0x7f0b58038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:36.589 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.571+0000 7f0b6dd9b700 1 --2- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0b580366b0 0x7f0b58038b60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b60009990 tx=0x7f0b60006e40 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:36.592 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.575+0000 7f0b5effd700 1 -- 172.21.15.104:0/3546328017 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0b64023300 con 0x7f0b700fcbd0 2024-02-19T18:53:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.840+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b7010ecc0 con 0x7f0b700fcbd0 2024-02-19T18:53:36.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.841+0000 7f0b5effd700 1 -- 172.21.15.104:0/3546328017 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0b64047030 con 0x7f0b700fcbd0 2024-02-19T18:53:36.858 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:36.858 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.842+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0b580366b0 msgr2=0x7f0b58038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.842+0000 7f0b74ed1700 1 --2- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0b580366b0 0x7f0b58038b60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b60009990 tx=0x7f0b60006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.842+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 msgr2=0x7f0b7010ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:36.859 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.842+0000 7f0b74ed1700 1 --2- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 0x7f0b7010ced0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0b64009680 tx=0x7f0b64009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.843+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 shutdown_connections 2024-02-19T18:53:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.843+0000 7f0b74ed1700 1 --2- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0b580366b0 0x7f0b58038b60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.843+0000 7f0b74ed1700 1 --2- 172.21.15.104:0/3546328017 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0b700fcbd0 0x7f0b7010ced0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.843+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 >> 172.21.15.104:0/3546328017 conn(0x7f0b700f8290 msgr2=0x7f0b700f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.843+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 shutdown_connections 2024-02-19T18:53:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:36.843+0000 7f0b74ed1700 1 -- 172.21.15.104:0/3546328017 wait complete. 2024-02-19T18:53:36.860 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:37.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:36 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3546328017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:38.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:37 smithi077 ceph-mon[29247]: mgrmap e17: smithi077.yiiqpd(active, since 2s) 2024-02-19T18:53:38.989 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:38.990 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:39.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:39.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:39.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:39.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:39.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd/host:smithi077", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:53:39.575 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:39.623 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /etc/ceph/ceph.conf 2024-02-19T18:53:41.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: Updating smithi077:/etc/ceph/ceph.conf 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: Updating smithi104:/etc/ceph/ceph.conf 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-02-19T18:53:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:53:41.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.133+0000 7f9ba3435700 1 -- 172.21.15.104:0/3368793315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 msgr2=0x7f9b9c101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:41.150 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.133+0000 7f9ba3435700 1 --2- 172.21.15.104:0/3368793315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 0x7f9b9c101f30 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c0048c0 tx=0x7f9b8c009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.133+0000 7f9ba3435700 1 -- 172.21.15.104:0/3368793315 shutdown_connections 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.133+0000 7f9ba3435700 1 --2- 172.21.15.104:0/3368793315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 0x7f9b9c101f30 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.133+0000 7f9ba3435700 1 -- 172.21.15.104:0/3368793315 >> 172.21.15.104:0/3368793315 conn(0x7f9b9c072850 msgr2=0x7f9b9c072c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.133+0000 7f9ba3435700 1 -- 172.21.15.104:0/3368793315 shutdown_connections 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.133+0000 7f9ba3435700 1 -- 172.21.15.104:0/3368793315 wait complete. 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.133+0000 7f9ba3435700 1 Processor -- start 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.134+0000 7f9ba3435700 1 -- start start 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.134+0000 7f9ba3435700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 0x7f9b9c111df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:41.151 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.134+0000 7f9ba3435700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c112330 con 0x7f9b9c101b60 2024-02-19T18:53:41.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.134+0000 7f9ba11d1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 0x7f9b9c111df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:41.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.134+0000 7f9ba11d1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 0x7f9b9c111df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:47280/0 (socket says 172.21.15.104:47280) 2024-02-19T18:53:41.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.134+0000 7f9ba11d1700 1 -- 172.21.15.104:0/4079814798 learned_addr learned my addr 172.21.15.104:0/4079814798 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:41.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.134+0000 7f9ba11d1700 1 -- 172.21.15.104:0/4079814798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c005040 con 0x7f9b9c101b60 2024-02-19T18:53:41.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.135+0000 7f9ba11d1700 1 --2- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 0x7f9b9c111df0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009b80 tx=0x7f9b8c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:41.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.135+0000 7f9b91ffb700 1 -- 172.21.15.104:0/4079814798 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b8c00be50 con 0x7f9b9c101b60 2024-02-19T18:53:41.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.135+0000 7f9b91ffb700 1 -- 172.21.15.104:0/4079814798 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9b8c009620 con 0x7f9b9c101b60 2024-02-19T18:53:41.153 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.135+0000 7f9b91ffb700 1 -- 172.21.15.104:0/4079814798 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b8c01c5b0 con 0x7f9b9c101b60 2024-02-19T18:53:41.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.135+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c112530 con 0x7f9b9c101b60 2024-02-19T18:53:41.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.135+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c112a20 con 0x7f9b9c101b60 2024-02-19T18:53:41.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.136+0000 7f9b91ffb700 1 -- 172.21.15.104:0/4079814798 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f9b8c009790 con 0x7f9b9c101b60 2024-02-19T18:53:41.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.136+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c05f5c0 con 0x7f9b9c101b60 2024-02-19T18:53:41.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.136+0000 7f9b91ffb700 1 --2- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f9b88038b70 0x7f9b8803b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:41.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.136+0000 7f9b91ffb700 1 -- 172.21.15.104:0/4079814798 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9b8c049ce0 con 0x7f9b9c101b60 2024-02-19T18:53:41.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.136+0000 7f9ba09d0700 1 --2- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f9b88038b70 0x7f9b8803b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:41.154 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.137+0000 7f9ba09d0700 1 --2- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f9b88038b70 0x7f9b8803b020 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009990 tx=0x7f9b98006e40 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:41.158 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.141+0000 7f9b91ffb700 1 -- 172.21.15.104:0/4079814798 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f9b8c011940 con 0x7f9b9c101b60 2024-02-19T18:53:41.424 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.408+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b9c10d5a0 con 0x7f9b9c101b60 2024-02-19T18:53:41.425 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.408+0000 7f9b91ffb700 1 -- 172.21.15.104:0/4079814798 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f9b8c011370 con 0x7f9b9c101b60 2024-02-19T18:53:41.425 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:41.425 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:41.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f9b88038b70 msgr2=0x7f9b8803b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:41.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 --2- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f9b88038b70 0x7f9b8803b020 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9b98009990 tx=0x7f9b98006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:41.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 msgr2=0x7f9b9c111df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:41.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 --2- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 0x7f9b9c111df0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b8c009b80 tx=0x7f9b8c009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:41.427 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 shutdown_connections 2024-02-19T18:53:41.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 --2- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f9b88038b70 0x7f9b8803b020 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:41.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 --2- 172.21.15.104:0/4079814798 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f9b9c101b60 0x7f9b9c111df0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:41.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 >> 172.21.15.104:0/4079814798 conn(0x7f9b9c072850 msgr2=0x7f9b9c103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:41.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 shutdown_connections 2024-02-19T18:53:41.428 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:41.410+0000 7f9ba3435700 1 -- 172.21.15.104:0/4079814798 wait complete. 2024-02-19T18:53:41.428 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:42.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:41 smithi077 ceph-mon[29247]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:53:42.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:41 smithi077 ceph-mon[29247]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:53:42.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:41 smithi077 ceph-mon[29247]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:53:42.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:41 smithi077 ceph-mon[29247]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:53:42.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:41 smithi077 ceph-mon[29247]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T18:53:42.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:41 smithi077 ceph-mon[29247]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T18:53:42.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:41 smithi077 ceph-mon[29247]: Deploying daemon crash.smithi104 on smithi104 2024-02-19T18:53:42.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:41 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/4079814798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:43.455 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:43.456 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:43.958 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:53:45.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:44 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:45.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:44 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:45.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:44 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:45.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:44 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:45.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:44 smithi077 ceph-mon[29247]: Deploying daemon node-exporter.smithi104 on smithi104 2024-02-19T18:53:45.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:44 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:45.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.559+0000 7f11105ce700 1 -- 172.21.15.104:0/3643567494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 msgr2=0x7f1108101f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:45.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.559+0000 7f11105ce700 1 --2- 172.21.15.104:0/3643567494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 0x7f1108101f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f11040048c0 tx=0x7f1104009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:45.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.559+0000 7f11105ce700 1 -- 172.21.15.104:0/3643567494 shutdown_connections 2024-02-19T18:53:45.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.559+0000 7f11105ce700 1 --2- 172.21.15.104:0/3643567494 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 0x7f1108101f30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:45.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.559+0000 7f11105ce700 1 -- 172.21.15.104:0/3643567494 >> 172.21.15.104:0/3643567494 conn(0x7f11080720b0 msgr2=0x7f11080724b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:45.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.559+0000 7f11105ce700 1 -- 172.21.15.104:0/3643567494 shutdown_connections 2024-02-19T18:53:45.576 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.559+0000 7f11105ce700 1 -- 172.21.15.104:0/3643567494 wait complete. 2024-02-19T18:53:45.577 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.559+0000 7f11105ce700 1 Processor -- start 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.560+0000 7f11105ce700 1 -- start start 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.560+0000 7f11105ce700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 0x7f1108112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.560+0000 7f11105ce700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11081134a0 con 0x7f1108101b60 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.560+0000 7f110e36a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 0x7f1108112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.560+0000 7f110e36a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 0x7f1108112f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:47302/0 (socket says 172.21.15.104:47302) 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.560+0000 7f110e36a700 1 -- 172.21.15.104:0/4147996203 learned_addr learned my addr 172.21.15.104:0/4147996203 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.561+0000 7f110e36a700 1 -- 172.21.15.104:0/4147996203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1104005040 con 0x7f1108101b60 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.561+0000 7f110e36a700 1 --2- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 0x7f1108112f60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1104009330 tx=0x7f1104009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.561+0000 7f10feffd700 1 -- 172.21.15.104:0/4147996203 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1104004e60 con 0x7f1108101b60 2024-02-19T18:53:45.578 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.561+0000 7f10feffd700 1 -- 172.21.15.104:0/4147996203 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1104009630 con 0x7f1108101b60 2024-02-19T18:53:45.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.561+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1108113700 con 0x7f1108101b60 2024-02-19T18:53:45.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.561+0000 7f10feffd700 1 -- 172.21.15.104:0/4147996203 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f110401c5b0 con 0x7f1108101b60 2024-02-19T18:53:45.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.561+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1108113b70 con 0x7f1108101b60 2024-02-19T18:53:45.579 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.562+0000 7f10feffd700 1 -- 172.21.15.104:0/4147996203 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f110401c870 con 0x7f1108101b60 2024-02-19T18:53:45.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.563+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110805f5c0 con 0x7f1108101b60 2024-02-19T18:53:45.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.563+0000 7f10feffd700 1 --2- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f10f4036960 0x7f10f4038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:45.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.563+0000 7f10feffd700 1 -- 172.21.15.104:0/4147996203 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1104049e00 con 0x7f1108101b60 2024-02-19T18:53:45.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.563+0000 7f110db69700 1 --2- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f10f4036960 0x7f10f4038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:45.580 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.563+0000 7f110db69700 1 --2- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f10f4036960 0x7f10f4038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009990 tx=0x7f10f8006e40 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:45.584 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.568+0000 7f10feffd700 1 -- 172.21.15.104:0/4147996203 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1104023300 con 0x7f1108101b60 2024-02-19T18:53:45.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.838+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f110810e6c0 con 0x7f1108101b60 2024-02-19T18:53:45.855 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.838+0000 7f10feffd700 1 -- 172.21.15.104:0/4147996203 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1104011b60 con 0x7f1108101b60 2024-02-19T18:53:45.856 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:45.856 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:45.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.840+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f10f4036960 msgr2=0x7f10f4038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:45.857 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.840+0000 7f11105ce700 1 --2- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f10f4036960 0x7f10f4038e10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009990 tx=0x7f10f8006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.840+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 msgr2=0x7f1108112f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.840+0000 7f11105ce700 1 --2- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 0x7f1108112f60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1104009330 tx=0x7f1104009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.841+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 shutdown_connections 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.841+0000 7f11105ce700 1 --2- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f10f4036960 0x7f10f4038e10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.841+0000 7f11105ce700 1 --2- 172.21.15.104:0/4147996203 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1108101b60 0x7f1108112f60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.841+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 >> 172.21.15.104:0/4147996203 conn(0x7f11080720b0 msgr2=0x7f11080fb130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.841+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 shutdown_connections 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:45.841+0000 7f11105ce700 1 -- 172.21.15.104:0/4147996203 wait complete. 2024-02-19T18:53:45.858 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:46.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:45 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/4147996203' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:47.461 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:47.462 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:47.732 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:53:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:53:50.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.043+0000 7fe39d07b700 1 -- 172.21.15.104:0/2511784933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 msgr2=0x7fe398101ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:50.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.043+0000 7fe39d07b700 1 --2- 172.21.15.104:0/2511784933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 0x7fe398101ed0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe388009c60 tx=0x7fe388004d10 comp rx=0 tx=0).stop 2024-02-19T18:53:50.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.043+0000 7fe39d07b700 1 -- 172.21.15.104:0/2511784933 shutdown_connections 2024-02-19T18:53:50.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.043+0000 7fe39d07b700 1 --2- 172.21.15.104:0/2511784933 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 0x7fe398101ed0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:50.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.043+0000 7fe39d07b700 1 -- 172.21.15.104:0/2511784933 >> 172.21.15.104:0/2511784933 conn(0x7fe3980f7640 msgr2=0x7fe3980f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.043+0000 7fe39d07b700 1 -- 172.21.15.104:0/2511784933 shutdown_connections 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.043+0000 7fe39d07b700 1 -- 172.21.15.104:0/2511784933 wait complete. 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.044+0000 7fe39d07b700 1 Processor -- start 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.044+0000 7fe39d07b700 1 -- start start 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.044+0000 7fe39d07b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 0x7fe39810bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.044+0000 7fe39d07b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39810c160 con 0x7fe398101b00 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.044+0000 7fe396d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 0x7fe39810bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.044+0000 7fe396d9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 0x7fe39810bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:44470/0 (socket says 172.21.15.104:44470) 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.044+0000 7fe396d9d700 1 -- 172.21.15.104:0/12069809 learned_addr learned my addr 172.21.15.104:0/12069809 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:50.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.045+0000 7fe396d9d700 1 -- 172.21.15.104:0/12069809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388009940 con 0x7fe398101b00 2024-02-19T18:53:50.063 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.045+0000 7fe396d9d700 1 --2- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 0x7fe39810bc20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe388004e60 tx=0x7fe388004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.045+0000 7fe37f7fe700 1 -- 172.21.15.104:0/12069809 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe388007c80 con 0x7fe398101b00 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.045+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39810c360 con 0x7fe398101b00 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.045+0000 7fe37f7fe700 1 -- 172.21.15.104:0/12069809 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe38801d910 con 0x7fe398101b00 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.045+0000 7fe37f7fe700 1 -- 172.21.15.104:0/12069809 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe388012a00 con 0x7fe398101b00 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.045+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39810c850 con 0x7fe398101b00 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.046+0000 7fe37f7fe700 1 -- 172.21.15.104:0/12069809 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7fe388019040 con 0x7fe398101b00 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.046+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe398100d70 con 0x7fe398101b00 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.046+0000 7fe37f7fe700 1 --2- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe380036960 0x7fe380038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.046+0000 7fe37f7fe700 1 -- 172.21.15.104:0/12069809 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe38804a500 con 0x7fe398101b00 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.046+0000 7fe39659c700 1 --2- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe380036960 0x7fe380038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:50.064 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.047+0000 7fe39659c700 1 --2- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe380036960 0x7fe380038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009990 tx=0x7fe38c006e40 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:50.068 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.051+0000 7fe37f7fe700 1 -- 172.21.15.104:0/12069809 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe38801da80 con 0x7fe398101b00 2024-02-19T18:53:50.337 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.321+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe39805f5c0 con 0x7fe398101b00 2024-02-19T18:53:50.338 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.321+0000 7fe37f7fe700 1 -- 172.21.15.104:0/12069809 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe388017020 con 0x7fe398101b00 2024-02-19T18:53:50.338 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:50.338 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe380036960 msgr2=0x7fe380038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 --2- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe380036960 0x7fe380038e10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009990 tx=0x7fe38c006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:50.340 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 msgr2=0x7fe39810bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 --2- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 0x7fe39810bc20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe388004e60 tx=0x7fe388004d10 comp rx=0 tx=0).stop 2024-02-19T18:53:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 shutdown_connections 2024-02-19T18:53:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 --2- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe380036960 0x7fe380038e10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 --2- 172.21.15.104:0/12069809 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe398101b00 0x7fe39810bc20 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 >> 172.21.15.104:0/12069809 conn(0x7fe3980f7640 msgr2=0x7fe3980f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.323+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 shutdown_connections 2024-02-19T18:53:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:50.324+0000 7fe39d07b700 1 -- 172.21.15.104:0/12069809 wait complete. 2024-02-19T18:53:50.341 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:51.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.sdpzmi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.sdpzmi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: Deploying daemon mgr.smithi104.sdpzmi on smithi104 2024-02-19T18:53:51.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:51 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/12069809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:51.856 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:51.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:52.129 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.545+0000 7f656cf7b700 1 -- 172.21.15.104:0/3287961458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 msgr2=0x7f65680fdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.545+0000 7f656cf7b700 1 --2- 172.21.15.104:0/3287961458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 0x7f65680fdf90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f65580048c0 tx=0x7f6558009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.545+0000 7f656cf7b700 1 -- 172.21.15.104:0/3287961458 shutdown_connections 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.545+0000 7f656cf7b700 1 --2- 172.21.15.104:0/3287961458 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 0x7f65680fdf90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.545+0000 7f656cf7b700 1 -- 172.21.15.104:0/3287961458 >> 172.21.15.104:0/3287961458 conn(0x7f65680f7660 msgr2=0x7f65680f9a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.545+0000 7f656cf7b700 1 -- 172.21.15.104:0/3287961458 shutdown_connections 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.545+0000 7f656cf7b700 1 -- 172.21.15.104:0/3287961458 wait complete. 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.546+0000 7f656cf7b700 1 Processor -- start 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.546+0000 7f656cf7b700 1 -- start start 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.546+0000 7f656cf7b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 0x7f6568112fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:53.563 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.546+0000 7f656cf7b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6568113510 con 0x7f65680fdbc0 2024-02-19T18:53:53.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.546+0000 7f656659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 0x7f6568112fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:53.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.546+0000 7f656659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 0x7f6568112fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:44486/0 (socket says 172.21.15.104:44486) 2024-02-19T18:53:53.564 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.546+0000 7f656659c700 1 -- 172.21.15.104:0/1701018871 learned_addr learned my addr 172.21.15.104:0/1701018871 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:53.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.546+0000 7f656659c700 1 -- 172.21.15.104:0/1701018871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6558005040 con 0x7f65680fdbc0 2024-02-19T18:53:53.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.547+0000 7f656659c700 1 --2- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 0x7f6568112fd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6558009300 tx=0x7f6558009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:53.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.547+0000 7f6556ffd700 1 -- 172.21.15.104:0/1701018871 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6558004e60 con 0x7f65680fdbc0 2024-02-19T18:53:53.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.547+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6568113710 con 0x7f65680fdbc0 2024-02-19T18:53:53.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.547+0000 7f6556ffd700 1 -- 172.21.15.104:0/1701018871 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6558013420 con 0x7f65680fdbc0 2024-02-19T18:53:53.565 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.547+0000 7f6556ffd700 1 -- 172.21.15.104:0/1701018871 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f655801c5b0 con 0x7f65680fdbc0 2024-02-19T18:53:53.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.547+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6568113b80 con 0x7f65680fdbc0 2024-02-19T18:53:53.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.548+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656805f5c0 con 0x7f65680fdbc0 2024-02-19T18:53:53.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.548+0000 7f6556ffd700 1 -- 172.21.15.104:0/1701018871 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f6558013a90 con 0x7f65680fdbc0 2024-02-19T18:53:53.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.548+0000 7f6556ffd700 1 --2- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6550036960 0x7f6550038e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:53.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.548+0000 7f6556ffd700 1 -- 172.21.15.104:0/1701018871 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f6558049b60 con 0x7f65680fdbc0 2024-02-19T18:53:53.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.548+0000 7f6565d9b700 1 --2- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6550036960 0x7f6550038e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:53.566 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.549+0000 7f6565d9b700 1 --2- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6550036960 0x7f6550038e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f655c009990 tx=0x7f655c006e40 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:53.570 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.553+0000 7f6556ffd700 1 -- 172.21.15.104:0/1701018871 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6558011940 con 0x7f65680fdbc0 2024-02-19T18:53:53.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.824+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6568106f00 con 0x7f65680fdbc0 2024-02-19T18:53:53.842 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.825+0000 7f6556ffd700 1 -- 172.21.15.104:0/1701018871 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f6558011370 con 0x7f65680fdbc0 2024-02-19T18:53:53.842 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:53:53.843 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:49:36.693879Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-19T18:53:53.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.827+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6550036960 msgr2=0x7f6550038e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:53.844 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.827+0000 7f656cf7b700 1 --2- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6550036960 0x7f6550038e10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f655c009990 tx=0x7f655c006e40 comp rx=0 tx=0).stop 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.827+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 msgr2=0x7f6568112fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.827+0000 7f656cf7b700 1 --2- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 0x7f6568112fd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6558009300 tx=0x7f6558009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.828+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 shutdown_connections 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.828+0000 7f656cf7b700 1 --2- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6550036960 0x7f6550038e10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.828+0000 7f656cf7b700 1 --2- 172.21.15.104:0/1701018871 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f65680fdbc0 0x7f6568112fd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.828+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 >> 172.21.15.104:0/1701018871 conn(0x7f65680f7660 msgr2=0x7f6568101470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.828+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 shutdown_connections 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:53.828+0000 7f656cf7b700 1 -- 172.21.15.104:0/1701018871 wait complete. 2024-02-19T18:53:53.845 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-02-19T18:53:54.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:53 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1701018871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:53:55.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:55.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:53:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:53:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:53:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:55 smithi077 ceph-mon[29247]: Deploying daemon mon.smithi104 on smithi104 2024-02-19T18:53:55.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-02-19T18:53:55.756 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mon dump -f json 2024-02-19T18:53:57.169 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi104/config 2024-02-19T18:53:57.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:57 smithi077 ceph-mon[29247]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:59.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.041+0000 7f18d2120700 1 -- 172.21.15.104:0/1245581836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 msgr2=0x7f18cc101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:53:59.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.041+0000 7f18d2120700 1 --2- 172.21.15.104:0/1245581836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 0x7f18cc101fa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0048c0 tx=0x7f18bc009e30 comp rx=0 tx=0).stop 2024-02-19T18:53:59.058 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.041+0000 7f18d2120700 1 -- 172.21.15.104:0/1245581836 shutdown_connections 2024-02-19T18:53:59.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.041+0000 7f18d2120700 1 --2- 172.21.15.104:0/1245581836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 0x7f18cc101fa0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:53:59.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.041+0000 7f18d2120700 1 -- 172.21.15.104:0/1245581836 >> 172.21.15.104:0/1245581836 conn(0x7f18cc0f7760 msgr2=0x7f18cc0f9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:53:59.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.041+0000 7f18d2120700 1 -- 172.21.15.104:0/1245581836 shutdown_connections 2024-02-19T18:53:59.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.041+0000 7f18d2120700 1 -- 172.21.15.104:0/1245581836 wait complete. 2024-02-19T18:53:59.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.041+0000 7f18d2120700 1 Processor -- start 2024-02-19T18:53:59.059 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.042+0000 7f18d2120700 1 -- start start 2024-02-19T18:53:59.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.042+0000 7f18d2120700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 0x7f18cc111fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:59.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.042+0000 7f18d2120700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18cc1124f0 con 0x7f18cc101bd0 2024-02-19T18:53:59.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.042+0000 7f18cb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 0x7f18cc111fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:59.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.042+0000 7f18cb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 0x7f18cc111fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.104:47874/0 (socket says 172.21.15.104:47874) 2024-02-19T18:53:59.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.042+0000 7f18cb7fe700 1 -- 172.21.15.104:0/524512297 learned_addr learned my addr 172.21.15.104:0/524512297 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:53:59.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.043+0000 7f18cb7fe700 1 -- 172.21.15.104:0/524512297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc005040 con 0x7f18cc101bd0 2024-02-19T18:53:59.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.043+0000 7f18cb7fe700 1 --2- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 0x7f18cc111fb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009300 tx=0x7f18bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:59.060 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.043+0000 7f18b3fff700 1 -- 172.21.15.104:0/524512297 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18bc004e60 con 0x7f18cc101bd0 2024-02-19T18:53:59.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.043+0000 7f18b3fff700 1 -- 172.21.15.104:0/524512297 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18bc013420 con 0x7f18cc101bd0 2024-02-19T18:53:59.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.043+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18cc1126f0 con 0x7f18cc101bd0 2024-02-19T18:53:59.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.043+0000 7f18b3fff700 1 -- 172.21.15.104:0/524512297 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18bc01c5b0 con 0x7f18cc101bd0 2024-02-19T18:53:59.061 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.043+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18cc112be0 con 0x7f18cc101bd0 2024-02-19T18:53:59.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.044+0000 7f18b3fff700 1 -- 172.21.15.104:0/524512297 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 17) v1 ==== 43481+0+0 (secure 0 0 0) 0x7f18bc01c7f0 con 0x7f18cc101bd0 2024-02-19T18:53:59.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.044+0000 7f18b3fff700 1 --2- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f18b4036970 0x7f18b4038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:53:59.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.044+0000 7f18b3fff700 1 -- 172.21.15.104:0/524512297 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f18bc04ac90 con 0x7f18cc101bd0 2024-02-19T18:53:59.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.044+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18cc05f5c0 con 0x7f18cc101bd0 2024-02-19T18:53:59.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.044+0000 7f18caffd700 1 --2- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f18b4036970 0x7f18b4038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:53:59.062 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.045+0000 7f18caffd700 1 --2- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f18b4036970 0x7f18b4038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009990 tx=0x7f18c0006e40 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:53:59.066 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.049+0000 7f18b3fff700 1 -- 172.21.15.104:0/524512297 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f18bc023300 con 0x7f18cc101bd0 2024-02-19T18:53:59.178 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.161+0000 7f18b3fff700 1 -- 172.21.15.104:0/524512297 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18bc04aea0 con 0x7f18cc101bd0 2024-02-19T18:53:59.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:53:58 smithi077 ceph-mon[29247]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:53:59.342 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:53:59.325+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f18cc10d630 con 0x7f18cc101bd0 2024-02-19T18:54:04.215 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.198+0000 7f18b3fff700 1 -- 172.21.15.104:0/524512297 <== mon.0 v2:172.21.15.77:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7f18bc034090 con 0x7f18cc101bd0 2024-02-19T18:54:04.215 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-02-19T18:54:04.215 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","modified":"2024-02-19T18:53:59.159628Z","created":"2024-02-19T18:49:36.693879Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi077","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi104","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f18b4036970 msgr2=0x7f18b4038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 --2- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f18b4036970 0x7f18b4038e20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f18c0009990 tx=0x7f18c0006e40 comp rx=0 tx=0).stop 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 msgr2=0x7f18cc111fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 --2- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 0x7f18cc111fb0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009300 tx=0x7f18bc009e30 comp rx=0 tx=0).stop 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 shutdown_connections 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 --2- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f18b4036970 0x7f18b4038e20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 --2- 172.21.15.104:0/524512297 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f18cc101bd0 0x7f18cc111fb0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 >> 172.21.15.104:0/524512297 conn(0x7f18cc0f7760 msgr2=0x7f18cc0f9640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 shutdown_connections 2024-02-19T18:54:04.218 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:54:04.201+0000 7f18d2120700 1 -- 172.21.15.104:0/524512297 wait complete. 2024-02-19T18:54:04.219 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: mon.smithi077 calling monitor election 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/524512297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/4047541886' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/crt"}]: dispatch 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: mon.smithi104 calling monitor election 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: mon.smithi077 is new leader, mons smithi077,smithi104 in quorum (ranks 0,1) 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: monmap e2: 2 mons at {smithi077=[v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0],smithi104=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: fsmap 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: osdmap e5: 0 total, 0 up, 0 in 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: mgrmap e17: smithi077.yiiqpd(active, since 30s) 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: Standby manager daemon smithi104.sdpzmi started 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/4047541886' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: overall HEALTH_OK 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/4047541886' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/key"}]: dispatch 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/4047541886' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:04.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:04.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:04.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:05.057 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-19T18:54:05.057 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph config generate-minimal-conf 2024-02-19T18:54:05.332 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:54:05.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:05 smithi077 ceph-mon[29247]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:05.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:05 smithi077 ceph-mon[29247]: mgrmap e18: smithi077.yiiqpd(active, since 30s), standbys: smithi104.sdpzmi 2024-02-19T18:54:05.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi104.sdpzmi", "id": "smithi104.sdpzmi"}]: dispatch 2024-02-19T18:54:05.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:54:05.349 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.972+0000 7fe1f4d83700 1 -- 172.21.15.77:0/3250806739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0102c70 msgr2=0x7fe1f0103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.972+0000 7fe1f4d83700 1 --2- 172.21.15.77:0/3250806739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0102c70 0x7fe1f0103040 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe1e00048c0 tx=0x7fe1e0009e30 comp rx=0 tx=0).stop 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.972+0000 7fe1f4d83700 1 -- 172.21.15.77:0/3250806739 shutdown_connections 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.972+0000 7fe1f4d83700 1 --2- 172.21.15.77:0/3250806739 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0102c70 0x7fe1f0103040 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.972+0000 7fe1f4d83700 1 -- 172.21.15.77:0/3250806739 >> 172.21.15.77:0/3250806739 conn(0x7fe1f00720e0 msgr2=0x7fe1f00724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.972+0000 7fe1f4d83700 1 -- 172.21.15.77:0/3250806739 shutdown_connections 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.972+0000 7fe1f4d83700 1 -- 172.21.15.77:0/3250806739 wait complete. 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.972+0000 7fe1f4d83700 1 Processor -- start 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1f4d83700 1 -- start start 2024-02-19T18:54:07.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1f4d83700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 0x7fe1f0113140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1f4d83700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0113680 0x7fe1f010d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1f4d83700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0113c90 con 0x7fe1f0113680 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1f4d83700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f0113e00 con 0x7fe1f0102c70 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1edd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0113680 0x7fe1f010d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1edd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0113680 0x7fe1f010d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41954/0 (socket says 172.21.15.77:41954) 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1edd9b700 1 -- 172.21.15.77:0/271963841 learned_addr learned my addr 172.21.15.77:0/271963841 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1ee59c700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 0x7fe1f0113140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1ee59c700 1 -- 172.21.15.77:0/271963841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 msgr2=0x7fe1f0113140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1ee59c700 1 -- 172.21.15.77:0/271963841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 msgr2=0x7fe1f0113140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1ee59c700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 0x7fe1f0113140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1ee59c700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 0x7fe1f0113140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1edd9b700 1 -- 172.21.15.77:0/271963841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 msgr2=0x7fe1f0113140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1edd9b700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 0x7fe1f0113140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1edd9b700 1 -- 172.21.15.77:0/271963841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e0005040 con 0x7fe1f0113680 2024-02-19T18:54:07.013 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.973+0000 7fe1edd9b700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0113680 0x7fe1f010d020 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400b790 tx=0x7fe1e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.974+0000 7fe1deffd700 1 -- 172.21.15.77:0/271963841 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e4010870 con 0x7fe1f0113680 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.974+0000 7fe1deffd700 1 -- 172.21.15.77:0/271963841 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe1e400f3e0 con 0x7fe1f0113680 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.974+0000 7fe1deffd700 1 -- 172.21.15.77:0/271963841 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1e4010ea0 con 0x7fe1f0113680 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.974+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f010d680 con 0x7fe1f0113680 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.974+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f010dbc0 con 0x7fe1f0113680 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.975+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f010de90 con 0x7fe1f0113680 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.975+0000 7fe1deffd700 1 -- 172.21.15.77:0/271963841 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fe1e400f550 con 0x7fe1f0113680 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.976+0000 7fe1deffd700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1d8069370 0x7fe1d806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.976+0000 7fe1ee59c700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1d8069370 0x7fe1d806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.976+0000 7fe1deffd700 1 -- 172.21.15.77:0/271963841 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe1e40865b0 con 0x7fe1f0113680 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.976+0000 7fe1ee59c700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1d8069370 0x7fe1d806b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000b3b0 tx=0x7fe1e0009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:07.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:06.980+0000 7fe1deffd700 1 -- 172.21.15.77:0/271963841 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe1f010de90 con 0x7fe1f0113680 2024-02-19T18:54:07.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.181+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fe1f005f5c0 con 0x7fe1f0113680 2024-02-19T18:54:07.199 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.182+0000 7fe1deffd700 1 -- 172.21.15.77:0/271963841 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7fe1e4014100 con 0x7fe1f0113680 2024-02-19T18:54:07.199 INFO:teuthology.orchestra.run.smithi077.stdout:# minimal ceph.conf for 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:54:07.199 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-02-19T18:54:07.199 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 6778a95a-cf57-11ee-95bb-87774f69a715 2024-02-19T18:54:07.199 INFO:teuthology.orchestra.run.smithi077.stdout: mon_host = [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-02-19T18:54:07.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1d8069370 msgr2=0x7fe1d806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:07.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1d8069370 0x7fe1d806b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000b3b0 tx=0x7fe1e0009e30 comp rx=0 tx=0).stop 2024-02-19T18:54:07.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0113680 msgr2=0x7fe1f010d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:07.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0113680 0x7fe1f010d020 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe1e400b790 tx=0x7fe1e4009250 comp rx=0 tx=0).stop 2024-02-19T18:54:07.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 shutdown_connections 2024-02-19T18:54:07.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1d8069370 0x7fe1d806b820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:07.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe1f0102c70 0x7fe1f0113140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:07.201 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 --2- 172.21.15.77:0/271963841 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe1f0113680 0x7fe1f010d020 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:07.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.184+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 >> 172.21.15.77:0/271963841 conn(0x7fe1f00720e0 msgr2=0x7fe1f0104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:07.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.185+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 shutdown_connections 2024-02-19T18:54:07.202 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:07.185+0000 7fe1f4d83700 1 -- 172.21.15.77:0/271963841 wait complete. 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: Updating smithi077:/etc/ceph/ceph.conf 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: Updating smithi104:/etc/ceph/ceph.conf 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:07.507 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:07.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:07.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:07.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:54:07.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:54:07.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:07.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:07 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/271963841' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:07.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:07 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/271963841' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:07.822 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-19T18:54:07.822 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:54:07.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.conf 2024-02-19T18:54:07.851 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:54:07.851 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:54:07.918 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:54:07.918 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-02-19T18:54:07.953 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:54:07.953 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-19T18:54:08.024 INFO:tasks.cephadm:Deploying OSDs... 2024-02-19T18:54:08.024 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:54:08.024 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T18:54:08.039 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T18:54:08.039 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-02-19T18:54:08.094 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T18:54:08.095 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:54:08.095 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-02-19T18:54:08.095 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:54:08.095 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:54:08.095 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-02-19 18:53:25.898897835 +0000 2024-02-19T18:54:08.095 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-02-19 18:44:32.214171305 +0000 2024-02-19T18:54:08.095 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-02-19 18:44:32.214171305 +0000 2024-02-19T18:54:08.095 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-02-19 18:44:32.214171305 +0000 2024-02-19T18:54:08.095 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T18:54:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-02-19T18:54:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-02-19T18:54:08.160 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.00022099 s, 2.3 MB/s 2024-02-19T18:54:08.162 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T18:54:08.218 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-02-19T18:54:08.273 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T18:54:08.274 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:54:08.274 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-19T18:54:08.274 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:54:08.274 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:54:08.274 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-02-19 18:53:25.899897813 +0000 2024-02-19T18:54:08.274 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-02-19 18:44:32.643161781 +0000 2024-02-19T18:54:08.274 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-02-19 18:44:32.643161781 +0000 2024-02-19T18:54:08.274 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-02-19 18:44:32.643161781 +0000 2024-02-19T18:54:08.274 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T18:54:08.338 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-02-19T18:54:08.338 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-02-19T18:54:08.339 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000193189 s, 2.7 MB/s 2024-02-19T18:54:08.340 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T18:54:08.396 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-02-19T18:54:08.450 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T18:54:08.451 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:54:08.451 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-02-19T18:54:08.451 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:54:08.451 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:54:08.451 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-02-19 18:53:25.899897813 +0000 2024-02-19T18:54:08.451 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-02-19 18:44:33.047152811 +0000 2024-02-19T18:54:08.451 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-02-19 18:44:33.047152811 +0000 2024-02-19T18:54:08.451 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-02-19 18:44:33.047152811 +0000 2024-02-19T18:54:08.451 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T18:54:08.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:08 smithi077 ceph-mon[29247]: Reconfiguring mon.smithi077 (unknown last config time)... 2024-02-19T18:54:08.517 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:08 smithi077 ceph-mon[29247]: Reconfiguring daemon mon.smithi077 on smithi077 2024-02-19T18:54:08.520 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-02-19T18:54:08.520 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-02-19T18:54:08.520 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000172636 s, 3.0 MB/s 2024-02-19T18:54:08.521 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T18:54:08.577 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-02-19 18:53:25.899897813 +0000 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-02-19 18:44:33.537141932 +0000 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-02-19 18:44:33.537141932 +0000 2024-02-19T18:54:08.633 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-02-19 18:44:33.537141932 +0000 2024-02-19T18:54:08.634 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T18:54:08.699 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-02-19T18:54:08.699 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-02-19T18:54:08.700 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000209027 s, 2.4 MB/s 2024-02-19T18:54:08.701 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T18:54:08.756 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:54:08.756 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-02-19T18:54:08.775 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-19T18:54:08.776 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-02-19T18:54:08.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:08 smithi104 ceph-mon[42320]: Reconfiguring mon.smithi077 (unknown last config time)... 2024-02-19T18:54:08.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:08 smithi104 ceph-mon[42320]: Reconfiguring daemon mon.smithi077 on smithi077 2024-02-19T18:54:08.828 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-19T18:54:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:54:08.829 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-02-19T18:54:08.829 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:54:08.829 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:54:08.829 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-02-19 18:53:57.415661249 +0000 2024-02-19T18:54:08.829 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-02-19 18:44:32.181675010 +0000 2024-02-19T18:54:08.829 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-02-19 18:44:32.181675010 +0000 2024-02-19T18:54:08.829 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-02-19 18:44:32.181675010 +0000 2024-02-19T18:54:08.829 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-19T18:54:08.898 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-02-19T18:54:08.898 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-02-19T18:54:08.898 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000239297 s, 2.1 MB/s 2024-02-19T18:54:08.899 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-19T18:54:08.958 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-02-19T18:54:09.016 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-19T18:54:09.016 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:54:09.016 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-19T18:54:09.016 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:54:09.016 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:54:09.016 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-02-19 18:53:57.416661237 +0000 2024-02-19T18:54:09.017 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-02-19 18:44:32.693668635 +0000 2024-02-19T18:54:09.017 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-02-19 18:44:32.693668635 +0000 2024-02-19T18:54:09.017 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-02-19 18:44:32.693668635 +0000 2024-02-19T18:54:09.017 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-19T18:54:09.085 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-02-19T18:54:09.085 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-02-19T18:54:09.085 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000240981 s, 2.1 MB/s 2024-02-19T18:54:09.086 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-19T18:54:09.143 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-02-19T18:54:09.199 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-19T18:54:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:54:09.200 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-19T18:54:09.200 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:54:09.200 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:54:09.200 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-02-19 18:53:57.416661237 +0000 2024-02-19T18:54:09.200 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-02-19 18:44:33.268661475 +0000 2024-02-19T18:54:09.200 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-02-19 18:44:33.268661475 +0000 2024-02-19T18:54:09.200 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-02-19 18:44:33.268661475 +0000 2024-02-19T18:54:09.200 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-19T18:54:09.266 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-02-19T18:54:09.266 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-02-19T18:54:09.267 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000181634 s, 2.8 MB/s 2024-02-19T18:54:09.268 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-19T18:54:09.325 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-02-19 18:53:57.416661237 +0000 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-02-19 18:44:33.791654962 +0000 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-02-19 18:44:33.791654962 +0000 2024-02-19T18:54:09.381 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-02-19 18:44:33.791654962 +0000 2024-02-19T18:54:09.382 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-19T18:54:09.447 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-02-19T18:54:09.448 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-02-19T18:54:09.448 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000191039 s, 2.7 MB/s 2024-02-19T18:54:09.449 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-19T18:54:09.505 INFO:tasks.cephadm:Deploying osd.0 on smithi077 with /dev/vg_nvme/lv_4... 2024-02-19T18:54:09.505 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-19T18:54:09.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:09 smithi104 ceph-mon[42320]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:09.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:09 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:09.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:09 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:09.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:09 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.yiiqpd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:54:09.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:09 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:54:09.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:09 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:09.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:09 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:09.618 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:09 smithi077 ceph-mon[29247]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:09.618 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:09 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:09.618 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:09 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:09.618 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:09 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.yiiqpd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:54:09.619 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:09 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:54:09.619 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:09 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:09.619 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:09 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:09.860 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:54:10.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:10 smithi077 ceph-mon[29247]: Reconfiguring mgr.smithi077.yiiqpd (unknown last config time)... 2024-02-19T18:54:10.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:10 smithi077 ceph-mon[29247]: Reconfiguring daemon mgr.smithi077.yiiqpd on smithi077 2024-02-19T18:54:10.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:10 smithi104 ceph-mon[42320]: Reconfiguring mgr.smithi077.yiiqpd (unknown last config time)... 2024-02-19T18:54:10.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:10 smithi104 ceph-mon[42320]: Reconfiguring daemon mgr.smithi077.yiiqpd on smithi077 2024-02-19T18:54:11.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:11 smithi077 ceph-mon[29247]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:11.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:11 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:11.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:11 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:11.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:11 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:54:11.398 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:11 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:11.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:11 smithi104 ceph-mon[42320]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:11.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:11 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:11.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:11 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:11.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:11 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi077", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:54:11.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:11 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:12.333 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-19T18:54:12.333 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-19T18:54:12.333 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-02-19T18:54:12.333 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-02-19T18:54:12.333 INFO:teuthology.orchestra.run.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114203 s, 918 MB/s 2024-02-19T18:54:12.333 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-02-19T18:54:12.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:12 smithi077 ceph-mon[29247]: Reconfiguring crash.smithi077 (monmap changed)... 2024-02-19T18:54:12.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:12 smithi077 ceph-mon[29247]: Reconfiguring daemon crash.smithi077 on smithi077 2024-02-19T18:54:12.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:12 smithi104 ceph-mon[42320]: Reconfiguring crash.smithi077 (monmap changed)... 2024-02-19T18:54:12.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:12 smithi104 ceph-mon[42320]: Reconfiguring daemon crash.smithi077 on smithi077 2024-02-19T18:54:12.958 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_4 2024-02-19T18:54:13.445 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:54:13.585 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:13 smithi077 ceph-mon[29247]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:13.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:13 smithi104 ceph-mon[42320]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:14.654 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:14 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:14.654 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:14 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:14.654 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:14 smithi077 ceph-mon[29247]: Reconfiguring alertmanager.smithi077 (dependencies changed)... 2024-02-19T18:54:14.654 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:14 smithi077 ceph-mon[29247]: Reconfiguring daemon alertmanager.smithi077 on smithi077 2024-02-19T18:54:14.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:14 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:14.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:14 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:14.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:14 smithi104 ceph-mon[42320]: Reconfiguring alertmanager.smithi077 (dependencies changed)... 2024-02-19T18:54:14.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:14 smithi104 ceph-mon[42320]: Reconfiguring daemon alertmanager.smithi077 on smithi077 2024-02-19T18:54:15.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:15 smithi077 ceph-mon[29247]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.575+0000 7ff6b1639700 1 -- 172.21.15.77:0/1557012067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 msgr2=0x7ff6ac0fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.575+0000 7ff6b1639700 1 --2- 172.21.15.77:0/1557012067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 0x7ff6ac0fc250 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4009ce0 tx=0x7ff6a4004d10 comp rx=0 tx=0).stop 2024-02-19T18:54:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.575+0000 7ff6b1639700 1 -- 172.21.15.77:0/1557012067 shutdown_connections 2024-02-19T18:54:15.592 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.575+0000 7ff6b1639700 1 --2- 172.21.15.77:0/1557012067 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6ac0fcfb0 0x7ff6ac0fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.575+0000 7ff6b1639700 1 --2- 172.21.15.77:0/1557012067 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 0x7ff6ac0fc250 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.575+0000 7ff6b1639700 1 -- 172.21.15.77:0/1557012067 >> 172.21.15.77:0/1557012067 conn(0x7ff6ac0f7520 msgr2=0x7ff6ac0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.575+0000 7ff6b1639700 1 -- 172.21.15.77:0/1557012067 shutdown_connections 2024-02-19T18:54:15.593 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.575+0000 7ff6b1639700 1 -- 172.21.15.77:0/1557012067 wait complete. 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.576+0000 7ff6b1639700 1 Processor -- start 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.576+0000 7ff6b1639700 1 -- start start 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6b1639700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 0x7ff6ac10fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6b1639700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6ac0fcfb0 0x7ff6ac110260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6b1639700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac110fc0 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6b1639700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ac111130 con 0x7ff6ac0fcfb0 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6aaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 0x7ff6ac10fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6aaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 0x7ff6ac10fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:41978/0 (socket says 172.21.15.77:41978) 2024-02-19T18:54:15.595 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6aaffd700 1 -- 172.21.15.77:0/2231771373 learned_addr learned my addr 172.21.15.77:0/2231771373 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6aaffd700 1 -- 172.21.15.77:0/2231771373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6ac0fcfb0 msgr2=0x7ff6ac110260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6aaffd700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6ac0fcfb0 0x7ff6ac110260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6aaffd700 1 -- 172.21.15.77:0/2231771373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a4009940 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6aaffd700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 0x7ff6ac10fd20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4015040 tx=0x7ff6a4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6a37fe700 1 -- 172.21.15.77:0/2231771373 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6a400e040 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6a37fe700 1 -- 172.21.15.77:0/2231771373 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6a40128e0 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ac1113b0 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6a37fe700 1 -- 172.21.15.77:0/2231771373 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff6a401c940 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.577+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ac1118f0 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.578+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ac04bed0 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.579+0000 7ff6a37fe700 1 -- 172.21.15.77:0/2231771373 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7ff6a4012410 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.579+0000 7ff6a37fe700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff698069210 0x7ff69806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:15.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.579+0000 7ff6a37fe700 1 -- 172.21.15.77:0/2231771373 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff6a4087e20 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.579+0000 7ff6aa7fc700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff698069210 0x7ff69806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:15.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.580+0000 7ff6aa7fc700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff698069210 0x7ff69806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff694006fd0 tx=0x7ff694006db0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:15.600 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.584+0000 7ff6a37fe700 1 -- 172.21.15.77:0/2231771373 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff6a4058c40 con 0x7ff6ac0fbe40 2024-02-19T18:54:15.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:15.778+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6ac066660 con 0x7ff698069210 2024-02-19T18:54:15.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:15 smithi104 ceph-mon[42320]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:16.447 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:16 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:54:16.447 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:16 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:54:16.447 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:16 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:16.712 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:16 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:54:16.713 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:16 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:54:16.713 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:16 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:17.691 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:17 smithi077 ceph-mon[29247]: from='client.14302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:54:17.691 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:17 smithi077 ceph-mon[29247]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:17.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:17 smithi104 ceph-mon[42320]: from='client.14302 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:54:17.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:17 smithi104 ceph-mon[42320]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:19.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:18 smithi077 ceph-mon[29247]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:19.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:18 smithi104 ceph-mon[42320]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:20.051 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:54:20.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:54:21.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:21 smithi077 ceph-mon[29247]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:21.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:21 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:21.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:21 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:21.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:21 smithi077 ceph-mon[29247]: Reconfiguring grafana.smithi077 (dependencies changed)... 2024-02-19T18:54:21.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:21 smithi077 ceph-mon[29247]: Reconfiguring daemon grafana.smithi077 on smithi077 2024-02-19T18:54:21.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:21 smithi104 ceph-mon[42320]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:21.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:21 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:21.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:21 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:21.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:21 smithi104 ceph-mon[42320]: Reconfiguring grafana.smithi077 (dependencies changed)... 2024-02-19T18:54:21.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:21 smithi104 ceph-mon[42320]: Reconfiguring daemon grafana.smithi077 on smithi077 2024-02-19T18:54:22.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:22 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1706397144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbc9e22d-2cb7-4421-b6d8-0dac7325974f"}]: dispatch 2024-02-19T18:54:22.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:22 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1706397144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbc9e22d-2cb7-4421-b6d8-0dac7325974f"}]': finished 2024-02-19T18:54:22.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:22 smithi077 ceph-mon[29247]: osdmap e6: 1 total, 0 up, 1 in 2024-02-19T18:54:22.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:22 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:22.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:22 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4251101826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:54:22.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:22 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1706397144' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fbc9e22d-2cb7-4421-b6d8-0dac7325974f"}]: dispatch 2024-02-19T18:54:22.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:22 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1706397144' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fbc9e22d-2cb7-4421-b6d8-0dac7325974f"}]': finished 2024-02-19T18:54:22.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:22 smithi104 ceph-mon[42320]: osdmap e6: 1 total, 0 up, 1 in 2024-02-19T18:54:22.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:22 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:22.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:22 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/4251101826' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:54:24.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:23 smithi077 ceph-mon[29247]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:24.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:23 smithi104 ceph-mon[42320]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:25.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:25 smithi077 ceph-mon[29247]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:26.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:25 smithi104 ceph-mon[42320]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:27.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:27 smithi077 ceph-mon[29247]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:27.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:27 smithi104 ceph-mon[42320]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:29.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:28 smithi077 ceph-mon[29247]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:29.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:28 smithi104 ceph-mon[42320]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:31.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:31 smithi077 ceph-mon[29247]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:31.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:31 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:31.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:31 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:31.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:31 smithi104 ceph-mon[42320]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:31.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:31 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:31.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:31 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:32.300 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:32 smithi077 ceph-mon[29247]: Reconfiguring prometheus.smithi077 (dependencies changed)... 2024-02-19T18:54:32.300 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:32 smithi077 ceph-mon[29247]: Reconfiguring daemon prometheus.smithi077 on smithi077 2024-02-19T18:54:32.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:32 smithi104 ceph-mon[42320]: Reconfiguring prometheus.smithi077 (dependencies changed)... 2024-02-19T18:54:32.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:32 smithi104 ceph-mon[42320]: Reconfiguring daemon prometheus.smithi077 on smithi077 2024-02-19T18:54:33.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:33 smithi077 ceph-mon[29247]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:33.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T18:54:33.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:33.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:33 smithi077 ceph-mon[29247]: Deploying daemon osd.0 on smithi077 2024-02-19T18:54:33.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:33 smithi104 ceph-mon[42320]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:33.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-19T18:54:33.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:33.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:33 smithi104 ceph-mon[42320]: Deploying daemon osd.0 on smithi077 2024-02-19T18:54:34.270 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:54:34.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:54:35.436 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:35 smithi077 ceph-mon[29247]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:35.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:35 smithi104 ceph-mon[42320]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:37.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:37 smithi104 ceph-mon[42320]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:37.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:37.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:37.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:54:37.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:37 smithi077 ceph-mon[29247]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi104", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-02-19T18:54:37.501 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:38.458 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:38 smithi104 ceph-mon[42320]: Reconfiguring crash.smithi104 (monmap changed)... 2024-02-19T18:54:38.458 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:38 smithi104 ceph-mon[42320]: Reconfiguring daemon crash.smithi104 on smithi104 2024-02-19T18:54:38.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:38 smithi077 ceph-mon[29247]: Reconfiguring crash.smithi104 (monmap changed)... 2024-02-19T18:54:38.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:38 smithi077 ceph-mon[29247]: Reconfiguring daemon crash.smithi104 on smithi104 2024-02-19T18:54:39.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:39 smithi104 ceph-mon[42320]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:39.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:39 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:39.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:39 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:39.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:39 smithi104 ceph-mon[42320]: Reconfiguring mgr.smithi104.sdpzmi (monmap changed)... 2024-02-19T18:54:39.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:39 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.sdpzmi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:54:39.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:39 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:54:39.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:39 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:39.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:39 smithi104 ceph-mon[42320]: Reconfiguring daemon mgr.smithi104.sdpzmi on smithi104 2024-02-19T18:54:39.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:39 smithi077 ceph-mon[29247]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:39.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:39.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:39.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:39 smithi077 ceph-mon[29247]: Reconfiguring mgr.smithi104.sdpzmi (monmap changed)... 2024-02-19T18:54:39.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.sdpzmi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T18:54:39.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T18:54:39.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:39 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:39.728 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:39 smithi077 ceph-mon[29247]: Reconfiguring daemon mgr.smithi104.sdpzmi on smithi104 2024-02-19T18:54:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:40 smithi077 ceph-mon[29247]: Reconfiguring mon.smithi104 (monmap changed)... 2024-02-19T18:54:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:54:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:54:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:40 smithi077 ceph-mon[29247]: Reconfiguring daemon mon.smithi104 on smithi104 2024-02-19T18:54:41.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:40 smithi077 ceph-mon[29247]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:41.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:40 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:41.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:40 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:41.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:40 smithi104 ceph-mon[42320]: Reconfiguring mon.smithi104 (monmap changed)... 2024-02-19T18:54:41.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:40 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-19T18:54:41.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:40 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-19T18:54:41.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:40 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:41.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:40 smithi104 ceph-mon[42320]: Reconfiguring daemon mon.smithi104 on smithi104 2024-02-19T18:54:41.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:40 smithi104 ceph-mon[42320]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:42.425 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi077.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi077.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:54:42.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi077.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:54:42.427 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi077.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:54:42.427 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.427 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:54:42.427 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.427 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:42 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:54:42.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T18:54:42.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:54:42.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi077.front.sepia.ceph.com:9093"}]: dispatch 2024-02-19T18:54:42.442 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi077.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi077.front.sepia.ceph.com:3000"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi077.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi077.front.sepia.ceph.com:9095"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:42.443 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:42 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:43.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:43 smithi077 ceph-mon[29247]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:43.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:43 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:43.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:43 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:43.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:43 smithi104 ceph-mon[42320]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:43.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:43 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:43.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:43 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:45.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:45 smithi077 ceph-mon[29247]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:45.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:45 smithi104 ceph-mon[42320]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:46.229 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:46 smithi077 ceph-mon[29247]: from='osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T18:54:46.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.283+0000 7ff6a37fe700 1 -- 172.21.15.77:0/2231771373 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff6ac066660 con 0x7ff698069210 2024-02-19T18:54:46.300 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 0 on host 'smithi077' 2024-02-19T18:54:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff698069210 msgr2=0x7ff69806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff698069210 0x7ff69806b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff694006fd0 tx=0x7ff694006db0 comp rx=0 tx=0).stop 2024-02-19T18:54:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 msgr2=0x7ff6ac10fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 0x7ff6ac10fd20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4015040 tx=0x7ff6a4004d10 comp rx=0 tx=0).stop 2024-02-19T18:54:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 shutdown_connections 2024-02-19T18:54:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff698069210 0x7ff69806b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff6ac0fbe40 0x7ff6ac10fd20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:46.302 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 --2- 172.21.15.77:0/2231771373 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff6ac0fcfb0 0x7ff6ac110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:46.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 >> 172.21.15.77:0/2231771373 conn(0x7ff6ac0f7520 msgr2=0x7ff6ac065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:46.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 shutdown_connections 2024-02-19T18:54:46.303 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:46.285+0000 7ff6b1639700 1 -- 172.21.15.77:0/2231771373 wait complete. 2024-02-19T18:54:46.463 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:46 smithi104 ceph-mon[42320]: from='osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-19T18:54:47.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:47 smithi077 ceph-mon[29247]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:47.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:47 smithi077 ceph-mon[29247]: from='osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T18:54:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:47 smithi077 ceph-mon[29247]: osdmap e7: 1 total, 0 up, 1 in 2024-02-19T18:54:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:47 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:47 smithi077 ceph-mon[29247]: from='osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-02-19T18:54:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:47 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:47 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:47.359 DEBUG:teuthology.orchestra.run.smithi077:osd.0> sudo journalctl -f -n 0 -u ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.0.service 2024-02-19T18:54:47.361 INFO:tasks.cephadm:Deploying osd.1 on smithi077 with /dev/vg_nvme/lv_3... 2024-02-19T18:54:47.361 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-19T18:54:47.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:47 smithi104 ceph-mon[42320]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:47 smithi104 ceph-mon[42320]: from='osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-19T18:54:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:47 smithi104 ceph-mon[42320]: osdmap e7: 1 total, 0 up, 1 in 2024-02-19T18:54:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:47 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:47 smithi104 ceph-mon[42320]: from='osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-02-19T18:54:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:47 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:47 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:47.766 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:54:48.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:48 smithi077 ceph-mon[29247]: from='osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-02-19T18:54:48.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:48 smithi077 ceph-mon[29247]: osdmap e8: 1 total, 0 up, 1 in 2024-02-19T18:54:48.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:48 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:48.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:48 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:48.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:48 smithi104 ceph-mon[42320]: from='osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-02-19T18:54:48.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:48 smithi104 ceph-mon[42320]: osdmap e8: 1 total, 0 up, 1 in 2024-02-19T18:54:48.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:48 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:48.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:48 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:49.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: purged_snaps scrub starts 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: purged_snaps scrub ok 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] boot 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: osdmap e9: 1 total, 1 up, 1 in 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:54:49.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: purged_snaps scrub starts 2024-02-19T18:54:49.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: purged_snaps scrub ok 2024-02-19T18:54:49.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-19T18:54:49.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:49.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: osd.0 [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] boot 2024-02-19T18:54:49.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: osdmap e9: 1 total, 1 up, 1 in 2024-02-19T18:54:49.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T18:54:49.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:49.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:49.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:54:50.534 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-19T18:54:50.534 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-19T18:54:50.534 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-02-19T18:54:50.534 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-02-19T18:54:50.534 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112304 s, 934 MB/s 2024-02-19T18:54:50.535 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-02-19T18:54:51.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:50 smithi077 ceph-mon[29247]: osdmap e10: 1 total, 1 up, 1 in 2024-02-19T18:54:51.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:50 smithi077 ceph-mon[29247]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:51.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:50 smithi104 ceph-mon[42320]: osdmap e10: 1 total, 1 up, 1 in 2024-02-19T18:54:51.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:50 smithi104 ceph-mon[42320]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:51.608 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_3 2024-02-19T18:54:52.740 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:54:53.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:53 smithi077 ceph-mon[29247]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:53.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:53 smithi104 ceph-mon[42320]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 -- 172.21.15.77:0/551001844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 msgr2=0x7fafec0ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 --2- 172.21.15.77:0/551001844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 0x7fafec0ffe60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fafdc004940 tx=0x7fafdc009e30 comp rx=0 tx=0).stop 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 -- 172.21.15.77:0/551001844 shutdown_connections 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 --2- 172.21.15.77:0/551001844 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafec1003a0 0x7fafec102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 --2- 172.21.15.77:0/551001844 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 0x7fafec0ffe60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 -- 172.21.15.77:0/551001844 >> 172.21.15.77:0/551001844 conn(0x7fafec0f7520 msgr2=0x7fafec0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 -- 172.21.15.77:0/551001844 shutdown_connections 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 -- 172.21.15.77:0/551001844 wait complete. 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 Processor -- start 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.616+0000 7faff111e700 1 -- start start 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7faff111e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 0x7fafec10d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7faff111e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafec1003a0 0x7fafec10de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7faff111e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec10e4b0 con 0x7fafec0fda80 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7faff111e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafec10ecf0 con 0x7fafec1003a0 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafead9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 0x7fafec10d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafead9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 0x7fafec10d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58336/0 (socket says 172.21.15.77:58336) 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafead9d700 1 -- 172.21.15.77:0/713062298 learned_addr learned my addr 172.21.15.77:0/713062298 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafead9d700 1 -- 172.21.15.77:0/713062298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafec1003a0 msgr2=0x7fafec10de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafead9d700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafec1003a0 0x7fafec10de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafead9d700 1 -- 172.21.15.77:0/713062298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafdc005040 con 0x7fafec0fda80 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafead9d700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 0x7fafec10d8e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00b3b0 tx=0x7fafdc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafe3fff700 1 -- 172.21.15.77:0/713062298 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafdc00e040 con 0x7fafec0fda80 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafec10ef70 con 0x7fafec0fda80 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafe3fff700 1 -- 172.21.15.77:0/713062298 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fafdc00bd90 con 0x7fafec0fda80 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7fafe3fff700 1 -- 172.21.15.77:0/713062298 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fafdc01c630 con 0x7fafec0fda80 2024-02-19T18:54:54.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.617+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafec10f4b0 con 0x7fafec0fda80 2024-02-19T18:54:54.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.618+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafec107ad0 con 0x7fafec0fda80 2024-02-19T18:54:54.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.619+0000 7fafe3fff700 1 -- 172.21.15.77:0/713062298 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fafdc0139f0 con 0x7fafec0fda80 2024-02-19T18:54:54.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.619+0000 7fafe3fff700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fafd8069210 0x7fafd806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:54:54.637 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.619+0000 7fafe259c700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fafd8069210 0x7fafd806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:54:54.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.619+0000 7fafe3fff700 1 -- 172.21.15.77:0/713062298 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fafdc087840 con 0x7fafec0fda80 2024-02-19T18:54:54.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.620+0000 7fafe259c700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fafd8069210 0x7fafd806b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fafd4006fd0 tx=0x7fafd4006db0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:54:54.640 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.624+0000 7fafe3fff700 1 -- 172.21.15.77:0/713062298 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fafdc058530 con 0x7fafec0fda80 2024-02-19T18:54:54.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:54:54.816+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fafec10f800 con 0x7fafd8069210 2024-02-19T18:54:55.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:55.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:54:55.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:54:55.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:55.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:55.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:55.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:54:55.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:55.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:55.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:54:55.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:55.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:54:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:56.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:56 smithi077 ceph-mon[29247]: from='client.14320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:54:56.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:56 smithi077 ceph-mon[29247]: Detected new or changed devices on smithi077 2024-02-19T18:54:56.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:56 smithi077 ceph-mon[29247]: Adjusting osd_memory_target on smithi077 to 12864M 2024-02-19T18:54:56.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:56 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:54:56.509 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:56 smithi104 ceph-mon[42320]: from='client.14320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:54:56.509 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:56 smithi104 ceph-mon[42320]: Detected new or changed devices on smithi077 2024-02-19T18:54:56.509 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:56 smithi104 ceph-mon[42320]: Adjusting osd_memory_target on smithi077 to 12864M 2024-02-19T18:54:56.509 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:56 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:54:57.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:57 smithi077 ceph-mon[29247]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:57.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:57 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:57.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:57 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:57.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:57 smithi104 ceph-mon[42320]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:57.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:57 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:57.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:57 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:54:58.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:58 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4034482766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c544f75d-7ae5-4261-b55e-f2be62afb9a3"}]: dispatch 2024-02-19T18:54:58.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:58 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4034482766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c544f75d-7ae5-4261-b55e-f2be62afb9a3"}]': finished 2024-02-19T18:54:58.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:58 smithi077 ceph-mon[29247]: osdmap e11: 2 total, 1 up, 2 in 2024-02-19T18:54:58.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:58 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:54:58.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:58 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/4034482766' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c544f75d-7ae5-4261-b55e-f2be62afb9a3"}]: dispatch 2024-02-19T18:54:58.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:58 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/4034482766' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c544f75d-7ae5-4261-b55e-f2be62afb9a3"}]': finished 2024-02-19T18:54:58.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:58 smithi104 ceph-mon[42320]: osdmap e11: 2 total, 1 up, 2 in 2024-02-19T18:54:58.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:58 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:54:59.450 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:59 smithi077 ceph-mon[29247]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:59.450 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:54:59 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1181600490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:54:59.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:59 smithi104 ceph-mon[42320]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:54:59.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:54:59 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1181600490' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:55:01.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:01 smithi077 ceph-mon[29247]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:01.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:01 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:01.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:01 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:01.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:01 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:01.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:01 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:01.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:01 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:01.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:01 smithi104 ceph-mon[42320]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:01.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:01 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:01.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:01 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:01.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:01 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:01.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:01 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:01.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:01 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:03.784 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:03 smithi077 ceph-mon[29247]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:03.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:03 smithi104 ceph-mon[42320]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:04.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:04.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:05.635 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:05 smithi077 ceph-mon[29247]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:05.635 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T18:55:05.635 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:05.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:05 smithi104 ceph-mon[42320]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:05.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:05 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-19T18:55:05.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:05 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:06.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:06 smithi077 ceph-mon[29247]: Deploying daemon osd.1 on smithi077 2024-02-19T18:55:06.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:06 smithi104 ceph-mon[42320]: Deploying daemon osd.1 on smithi077 2024-02-19T18:55:07.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:07 smithi077 ceph-mon[29247]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:07.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:07 smithi104 ceph-mon[42320]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:08.965 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:08 smithi077 ceph-mon[29247]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:09.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:08 smithi104 ceph-mon[42320]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:11.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:11 smithi077 ceph-mon[29247]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:11.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:11 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:11.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:11 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:11.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:11 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:11.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:11 smithi104 ceph-mon[42320]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:11.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:11 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:11.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:11 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:11.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:11 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:13.448 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:13 smithi077 ceph-mon[29247]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:13.448 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:13 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:13.448 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:13 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:13.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:13 smithi104 ceph-mon[42320]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:13.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:13 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:13.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:13 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:13.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.775+0000 7fafe3fff700 1 -- 172.21.15.77:0/713062298 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fafec10f800 con 0x7fafd8069210 2024-02-19T18:55:13.793 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 1 on host 'smithi077' 2024-02-19T18:55:13.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fafd8069210 msgr2=0x7fafd806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:13.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fafd8069210 0x7fafd806b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fafd4006fd0 tx=0x7fafd4006db0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 msgr2=0x7fafec10d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:13.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 0x7fafec10d8e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00b3b0 tx=0x7fafdc009e30 comp rx=0 tx=0).stop 2024-02-19T18:55:13.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 shutdown_connections 2024-02-19T18:55:13.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fafd8069210 0x7fafd806b6c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fafec0fda80 0x7fafec10d8e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.795 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 --2- 172.21.15.77:0/713062298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fafec1003a0 0x7fafec10de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:13.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 >> 172.21.15.77:0/713062298 conn(0x7fafec0f7520 msgr2=0x7fafec0f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:13.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 shutdown_connections 2024-02-19T18:55:13.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:13.778+0000 7faff111e700 1 -- 172.21.15.77:0/713062298 wait complete. 2024-02-19T18:55:14.459 DEBUG:teuthology.orchestra.run.smithi077:osd.1> sudo journalctl -f -n 0 -u ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.1.service 2024-02-19T18:55:14.462 INFO:tasks.cephadm:Deploying osd.2 on smithi077 with /dev/vg_nvme/lv_2... 2024-02-19T18:55:14.462 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-19T18:55:15.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:14 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:15.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:14 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:15.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:14 smithi077 ceph-mon[29247]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:15.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:14 smithi077 ceph-mon[29247]: from='osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T18:55:15.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:14 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:15.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:14 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:15.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:14 smithi104 ceph-mon[42320]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:15.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:14 smithi104 ceph-mon[42320]: from='osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-19T18:55:15.577 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:55:16.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:15 smithi077 ceph-mon[29247]: from='osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T18:55:16.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:15 smithi077 ceph-mon[29247]: osdmap e12: 2 total, 1 up, 2 in 2024-02-19T18:55:16.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:15 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:16.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:15 smithi077 ceph-mon[29247]: from='osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-02-19T18:55:16.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T18:55:15.753+0000 7fb4d02b9700 -1 osd.1 0 waiting for initial osdmap 2024-02-19T18:55:16.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T18:55:15.757+0000 7fb4cd0b8700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:55:16.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:15 smithi104 ceph-mon[42320]: from='osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-19T18:55:16.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:15 smithi104 ceph-mon[42320]: osdmap e12: 2 total, 1 up, 2 in 2024-02-19T18:55:16.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:15 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:16.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:15 smithi104 ceph-mon[42320]: from='osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-02-19T18:55:17.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:16 smithi077 ceph-mon[29247]: from='osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-02-19T18:55:17.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:16 smithi077 ceph-mon[29247]: osdmap e13: 2 total, 1 up, 2 in 2024-02-19T18:55:17.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:16 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:17.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:16 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:17.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:16 smithi077 ceph-mon[29247]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:17.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:16 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:17.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:16 smithi104 ceph-mon[42320]: from='osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-02-19T18:55:17.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:16 smithi104 ceph-mon[42320]: osdmap e13: 2 total, 1 up, 2 in 2024-02-19T18:55:17.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:16 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:17.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:16 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:17.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:16 smithi104 ceph-mon[42320]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-02-19T18:55:17.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:16 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:17.953 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:17 smithi077 ceph-mon[29247]: purged_snaps scrub starts 2024-02-19T18:55:17.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:17 smithi077 ceph-mon[29247]: purged_snaps scrub ok 2024-02-19T18:55:17.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:17 smithi077 ceph-mon[29247]: osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] boot 2024-02-19T18:55:17.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:17 smithi077 ceph-mon[29247]: osdmap e14: 2 total, 2 up, 2 in 2024-02-19T18:55:17.954 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:17 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:18.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:17 smithi104 ceph-mon[42320]: purged_snaps scrub starts 2024-02-19T18:55:18.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:17 smithi104 ceph-mon[42320]: purged_snaps scrub ok 2024-02-19T18:55:18.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:17 smithi104 ceph-mon[42320]: osd.1 [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] boot 2024-02-19T18:55:18.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:17 smithi104 ceph-mon[42320]: osdmap e14: 2 total, 2 up, 2 in 2024-02-19T18:55:18.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:17 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T18:55:19.019 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-19T18:55:19.019 INFO:teuthology.orchestra.run.smithi077.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-19T18:55:19.019 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10+0 records in 2024-02-19T18:55:19.019 INFO:teuthology.orchestra.run.smithi077.stderr:10+0 records out 2024-02-19T18:55:19.019 INFO:teuthology.orchestra.run.smithi077.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132757 s, 790 MB/s 2024-02-19T18:55:19.019 INFO:teuthology.orchestra.run.smithi077.stderr:--> Zapping successful for: 2024-02-19T18:55:19.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:18 smithi077 ceph-mon[29247]: osdmap e15: 2 total, 2 up, 2 in 2024-02-19T18:55:19.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:18 smithi077 ceph-mon[29247]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:19.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:18 smithi104 ceph-mon[42320]: osdmap e15: 2 total, 2 up, 2 in 2024-02-19T18:55:19.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:18 smithi104 ceph-mon[42320]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:19.806 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi077:vg_nvme/lv_2 2024-02-19T18:55:20.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:20.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:20.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:20.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:20.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:20.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:21.057 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:20 smithi077 ceph-mon[29247]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:21.105 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:55:21.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:20 smithi104 ceph-mon[42320]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:23.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:23 smithi077 ceph-mon[29247]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:23.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:23 smithi104 ceph-mon[42320]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:23.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.977+0000 7f6cb1829700 1 -- 172.21.15.77:0/3223526999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 msgr2=0x7f6cac0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:23.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.977+0000 7f6cb1829700 1 --2- 172.21.15.77:0/3223526999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 0x7f6cac0fe550 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0004940 tx=0x7f6ca0009e30 comp rx=0 tx=0).stop 2024-02-19T18:55:23.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.977+0000 7f6cb1829700 1 -- 172.21.15.77:0/3223526999 shutdown_connections 2024-02-19T18:55:23.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.977+0000 7f6cb1829700 1 --2- 172.21.15.77:0/3223526999 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 0x7f6cac0fe550 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:23.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.977+0000 7f6cb1829700 1 --2- 172.21.15.77:0/3223526999 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cac0fcef0 0x7f6cac0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:23.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.977+0000 7f6cb1829700 1 -- 172.21.15.77:0/3223526999 >> 172.21.15.77:0/3223526999 conn(0x7f6cac0f8620 msgr2=0x7f6cac0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:23.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.977+0000 7f6cb1829700 1 -- 172.21.15.77:0/3223526999 shutdown_connections 2024-02-19T18:55:23.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.977+0000 7f6cb1829700 1 -- 172.21.15.77:0/3223526999 wait complete. 2024-02-19T18:55:23.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6cb1829700 1 Processor -- start 2024-02-19T18:55:23.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6cb1829700 1 -- start start 2024-02-19T18:55:23.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6cb1829700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cac0fcef0 0x7f6cac10d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6cb1829700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 0x7f6cac10def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6cb1829700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac10e580 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6cb1829700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac10edc0 con 0x7f6cac0fcef0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6caa7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 0x7f6cac10def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6caa7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 0x7f6cac10def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49648/0 (socket says 172.21.15.77:49648) 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6caa7fc700 1 -- 172.21.15.77:0/799132374 learned_addr learned my addr 172.21.15.77:0/799132374 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.978+0000 7f6caaffd700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cac0fcef0 0x7f6cac10d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6caa7fc700 1 -- 172.21.15.77:0/799132374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cac0fcef0 msgr2=0x7f6cac10d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6caa7fc700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cac0fcef0 0x7f6cac10d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6caa7fc700 1 -- 172.21.15.77:0/799132374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca0005040 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6caa7fc700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 0x7f6cac10def0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009fd0 tx=0x7f6ca0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6c937fe700 1 -- 172.21.15.77:0/799132374 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ca000e040 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac10f040 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6c937fe700 1 -- 172.21.15.77:0/799132374 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ca000bed0 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.997 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6c937fe700 1 -- 172.21.15.77:0/799132374 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ca0024900 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.979+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac10f580 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.980+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cac04bed0 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.981+0000 7f6c937fe700 1 -- 172.21.15.77:0/799132374 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7f6ca001bc70 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.981+0000 7f6c937fe700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c94069210 0x7f6c9406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.981+0000 7f6c937fe700 1 -- 172.21.15.77:0/799132374 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7f6ca008ba80 con 0x7f6cac0fe0e0 2024-02-19T18:55:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.981+0000 7f6caaffd700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c94069210 0x7f6c9406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:23.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.982+0000 7f6caaffd700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c94069210 0x7f6c9406b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0fcc30 tx=0x7f6c9c0091a0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:24.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:23.986+0000 7f6c937fe700 1 -- 172.21.15.77:0/799132374 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6ca005c510 con 0x7f6cac0fe0e0 2024-02-19T18:55:24.217 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:24.200+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cac10fa60 con 0x7f6c94069210 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: Detected new or changed devices on smithi077 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: Adjusting osd_memory_target on smithi077 to 6432M 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:25.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:25.012 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:25.012 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:25.012 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:25.012 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='client.14338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:25.012 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:55:25.012 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:55:25.012 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:25.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: Detected new or changed devices on smithi077 2024-02-19T18:55:25.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:25.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:25.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: Adjusting osd_memory_target on smithi077 to 6432M 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='client.14338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi077:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:55:25.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:27.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:27 smithi077 ceph-mon[29247]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:27.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:27 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/998248324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8463e27a-7421-42a1-8b87-7185b9f61766"}]: dispatch 2024-02-19T18:55:27.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:27 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/998248324' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8463e27a-7421-42a1-8b87-7185b9f61766"}]': finished 2024-02-19T18:55:27.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:27 smithi077 ceph-mon[29247]: osdmap e16: 3 total, 2 up, 3 in 2024-02-19T18:55:27.544 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:27 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:27.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:27 smithi104 ceph-mon[42320]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:27.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:27 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/998248324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8463e27a-7421-42a1-8b87-7185b9f61766"}]: dispatch 2024-02-19T18:55:27.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:27 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/998248324' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8463e27a-7421-42a1-8b87-7185b9f61766"}]': finished 2024-02-19T18:55:27.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:27 smithi104 ceph-mon[42320]: osdmap e16: 3 total, 2 up, 3 in 2024-02-19T18:55:27.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:27 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:28.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:28 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3123604797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:55:28.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:28 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3123604797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:55:29.307 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:29 smithi077 ceph-mon[29247]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:29.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:29 smithi104 ceph-mon[42320]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:30.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:30.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:30.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:30.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:30.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:30.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:30.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:30.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:30.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:30.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:31.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:31 smithi077 ceph-mon[29247]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:31.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:31 smithi104 ceph-mon[42320]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:33.656 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:33 smithi077 ceph-mon[29247]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:33.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:33 smithi104 ceph-mon[42320]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:34.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:34.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:35.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:35 smithi077 ceph-mon[29247]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:35.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:35 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T18:55:35.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:35 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:35.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:35 smithi104 ceph-mon[42320]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:35.811 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:35 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-19T18:55:35.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:35 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:36.807 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:36 smithi077 ceph-mon[29247]: Deploying daemon osd.2 on smithi077 2024-02-19T18:55:37.061 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:36 smithi104 ceph-mon[42320]: Deploying daemon osd.2 on smithi077 2024-02-19T18:55:37.872 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:37 smithi077 ceph-mon[29247]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:38.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:37 smithi104 ceph-mon[42320]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:39.300 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:38 smithi077 ceph-mon[29247]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:39.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:38 smithi104 ceph-mon[42320]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:41.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:41 smithi077 ceph-mon[29247]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:41.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:41.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:41.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:41.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:41 smithi104 ceph-mon[42320]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:41.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:41.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:41.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:43.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:43 smithi077 ceph-mon[29247]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:43.561 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:43 smithi104 ceph-mon[42320]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:43.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.963+0000 7f6c937fe700 1 -- 172.21.15.77:0/799132374 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6cac10fa60 con 0x7f6c94069210 2024-02-19T18:55:43.981 INFO:teuthology.orchestra.run.smithi077.stdout:Created osd(s) 2 on host 'smithi077' 2024-02-19T18:55:43.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c94069210 msgr2=0x7f6c9406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:43.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c94069210 0x7f6c9406b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6cac0fcc30 tx=0x7f6c9c0091a0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 msgr2=0x7f6cac10def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:43.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 0x7f6cac10def0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009fd0 tx=0x7f6ca0009e30 comp rx=0 tx=0).stop 2024-02-19T18:55:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 shutdown_connections 2024-02-19T18:55:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c94069210 0x7f6c9406b6c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6cac0fcef0 0x7f6cac10d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 --2- 172.21.15.77:0/799132374 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6cac0fe0e0 0x7f6cac10def0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 >> 172.21.15.77:0/799132374 conn(0x7f6cac0f8620 msgr2=0x7f6cac1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 shutdown_connections 2024-02-19T18:55:43.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:55:43.965+0000 7f6cb1829700 1 -- 172.21.15.77:0/799132374 wait complete. 2024-02-19T18:55:44.849 DEBUG:teuthology.orchestra.run.smithi077:osd.2> sudo journalctl -f -n 0 -u ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.2.service 2024-02-19T18:55:44.851 INFO:tasks.cephadm:Deploying osd.3 on smithi104 with /dev/vg_nvme/lv_4... 2024-02-19T18:55:44.851 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-19T18:55:45.110 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi104/config 2024-02-19T18:55:45.135 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:44 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:45.135 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:44 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:45.135 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:44 smithi077 ceph-mon[29247]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:45.135 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:44 smithi077 ceph-mon[29247]: from='osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T18:55:45.232 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:44 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:45.232 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:44 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:45.232 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:44 smithi104 ceph-mon[42320]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:45.232 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:44 smithi104 ceph-mon[42320]: from='osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-19T18:55:46.263 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:45 smithi104 ceph-mon[42320]: from='osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T18:55:46.263 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:45 smithi104 ceph-mon[42320]: osdmap e17: 3 total, 2 up, 3 in 2024-02-19T18:55:46.264 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:45 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:46.264 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:45 smithi104 ceph-mon[42320]: from='osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-02-19T18:55:46.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:45 smithi077 ceph-mon[29247]: from='osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-19T18:55:46.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:45 smithi077 ceph-mon[29247]: osdmap e17: 3 total, 2 up, 3 in 2024-02-19T18:55:46.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:45 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:46.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:45 smithi077 ceph-mon[29247]: from='osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]: dispatch 2024-02-19T18:55:46.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T18:55:45.943+0000 7f759ab62700 -1 osd.2 0 waiting for initial osdmap 2024-02-19T18:55:46.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T18:55:45.952+0000 7f759595d700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:55:46.939 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-19T18:55:46.940 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-19T18:55:46.940 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-02-19T18:55:46.940 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-02-19T18:55:46.940 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01242 s, 844 MB/s 2024-02-19T18:55:46.940 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-02-19T18:55:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:46 smithi077 ceph-mon[29247]: from='osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-02-19T18:55:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:46 smithi077 ceph-mon[29247]: osdmap e18: 3 total, 2 up, 3 in 2024-02-19T18:55:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:46 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:46 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:47.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:46 smithi077 ceph-mon[29247]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:47.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:46 smithi104 ceph-mon[42320]: from='osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi077", "root=default"]}]': finished 2024-02-19T18:55:47.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:46 smithi104 ceph-mon[42320]: osdmap e18: 3 total, 2 up, 3 in 2024-02-19T18:55:47.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:46 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:47.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:46 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:47.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:46 smithi104 ceph-mon[42320]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-02-19T18:55:47.590 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-02-19T18:55:47.846 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi104/config 2024-02-19T18:55:48.167 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:47 smithi104 ceph-mon[42320]: purged_snaps scrub starts 2024-02-19T18:55:48.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:47 smithi104 ceph-mon[42320]: purged_snaps scrub ok 2024-02-19T18:55:48.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:47 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:48.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:47 smithi104 ceph-mon[42320]: osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] boot 2024-02-19T18:55:48.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:47 smithi104 ceph-mon[42320]: osdmap e19: 3 total, 3 up, 3 in 2024-02-19T18:55:48.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:47 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:48.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:47 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:48.168 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:47 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:48.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:47 smithi077 ceph-mon[29247]: purged_snaps scrub starts 2024-02-19T18:55:48.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:47 smithi077 ceph-mon[29247]: purged_snaps scrub ok 2024-02-19T18:55:48.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:47 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:48.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:47 smithi077 ceph-mon[29247]: osd.2 [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] boot 2024-02-19T18:55:48.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:47 smithi077 ceph-mon[29247]: osdmap e19: 3 total, 3 up, 3 in 2024-02-19T18:55:48.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:47 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T18:55:48.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:47 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:48.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:47 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:49.331 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:49 smithi104 ceph-mon[42320]: osdmap e20: 3 total, 3 up, 3 in 2024-02-19T18:55:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:49 smithi104 ceph-mon[42320]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:49.332 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T18:55:49.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.332+0000 7fcc07ff0700 1 -- 172.21.15.104:0/3415383781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 msgr2=0x7fcc000fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:49.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.332+0000 7fcc07ff0700 1 --2- 172.21.15.104:0/3415383781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 0x7fcc000fc250 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009ce0 tx=0x7fcbf8004d10 comp rx=0 tx=0).stop 2024-02-19T18:55:49.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.332+0000 7fcc07ff0700 1 -- 172.21.15.104:0/3415383781 shutdown_connections 2024-02-19T18:55:49.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.332+0000 7fcc07ff0700 1 --2- 172.21.15.104:0/3415383781 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc000fcfb0 0x7fcc000fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:49.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.332+0000 7fcc07ff0700 1 --2- 172.21.15.104:0/3415383781 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 0x7fcc000fc250 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:49.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.332+0000 7fcc07ff0700 1 -- 172.21.15.104:0/3415383781 >> 172.21.15.104:0/3415383781 conn(0x7fcc000f7520 msgr2=0x7fcc000f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:55:49.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.332+0000 7fcc07ff0700 1 -- 172.21.15.104:0/3415383781 shutdown_connections 2024-02-19T18:55:49.349 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.332+0000 7fcc07ff0700 1 -- 172.21.15.104:0/3415383781 wait complete. 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc07ff0700 1 Processor -- start 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc07ff0700 1 -- start start 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc07ff0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 0x7fcc000fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc07ff0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc000fcfb0 0x7fcc000fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc07ff0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000fe980 con 0x7fcc000fcfb0 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc07ff0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc000feaf0 con 0x7fcc000fbe40 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc05d8c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 0x7fcc000fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc05d8c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 0x7fcc000fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:52948/0 (socket says 172.21.15.104:52948) 2024-02-19T18:55:49.351 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc05d8c700 1 -- 172.21.15.104:0/2089412580 learned_addr learned my addr 172.21.15.104:0/2089412580 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc0558b700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc000fcfb0 0x7fcc000fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc05d8c700 1 -- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc000fcfb0 msgr2=0x7fcc000fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc05d8c700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc000fcfb0 0x7fcc000fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc05d8c700 1 -- 172.21.15.104:0/2089412580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf8009940 con 0x7fcc000fbe40 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.333+0000 7fcc05d8c700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 0x7fcc000fff00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8015040 tx=0x7fcbf8004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.334+0000 7fcbfe7fc700 1 -- 172.21.15.104:0/2089412580 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbf800e040 con 0x7fcc000fbe40 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.334+0000 7fcbfe7fc700 1 -- 172.21.15.104:0/2089412580 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcbf80128e0 con 0x7fcc000fbe40 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.334+0000 7fcbfe7fc700 1 -- 172.21.15.104:0/2089412580 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcbf801c760 con 0x7fcc000fbe40 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.334+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc000fed70 con 0x7fcc000fbe40 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.334+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc00115a40 con 0x7fcc000fbe40 2024-02-19T18:55:49.352 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.335+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0004bed0 con 0x7fcc000fbe40 2024-02-19T18:55:49.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.335+0000 7fcbfe7fc700 1 -- 172.21.15.104:0/2089412580 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 18) v1 ==== 86565+0+0 (secure 0 0 0) 0x7fcbf8012a50 con 0x7fcc000fbe40 2024-02-19T18:55:49.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.335+0000 7fcbfe7fc700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcbe8069210 0x7fcbe806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:55:49.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.336+0000 7fcbfe7fc700 1 -- 172.21.15.104:0/2089412580 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7fcbf80883f0 con 0x7fcc000fbe40 2024-02-19T18:55:49.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.336+0000 7fcc0558b700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcbe8069210 0x7fcbe806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:55:49.353 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.336+0000 7fcc0558b700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcbe8069210 0x7fcbe806b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009fd0 tx=0x7fcbf400b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:55:49.356 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.340+0000 7fcbfe7fc700 1 -- 172.21.15.104:0/2089412580 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcbf8058b50 con 0x7fcc000fbe40 2024-02-19T18:55:49.549 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:49.533+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc00066660 con 0x7fcbe8069210 2024-02-19T18:55:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:49 smithi077 ceph-mon[29247]: osdmap e20: 3 total, 3 up, 3 in 2024-02-19T18:55:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:49 smithi077 ceph-mon[29247]: pgmap v75: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:49.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-02-19T18:55:50.210 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:50 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-19T18:55:50.210 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:50 smithi104 ceph-mon[42320]: osdmap e21: 3 total, 3 up, 3 in 2024-02-19T18:55:50.210 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:50 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:55:50.211 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:50 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:50.211 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:50 smithi104 ceph-mon[42320]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:50.211 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:50 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:55:50.211 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:50 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:55:50.211 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:50 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:50.257 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:50 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-02-19T18:55:50.257 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:50 smithi077 ceph-mon[29247]: osdmap e21: 3 total, 3 up, 3 in 2024-02-19T18:55:50.258 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:50 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-19T18:55:50.258 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:50 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:55:50.258 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:50 smithi077 ceph-mon[29247]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:55:50.258 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:50 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:55:50.258 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:50 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:55:50.258 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:50 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: osdmap e22: 3 total, 3 up, 3 in 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: Detected new or changed devices on smithi077 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: Adjusting osd_memory_target on smithi077 to 4288M 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:51.151 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:51.152 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:51.152 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:51 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:51.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 18:55:50 smithi077 sudo[67510]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:55:51.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 18:55:50 smithi077 sudo[67510]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:51.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 18:55:50 smithi077 sudo[67510]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:51.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 18:55:50 smithi077 sudo[67510]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: osdmap e22: 3 total, 3 up, 3 in 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: Detected new or changed devices on smithi077 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: Adjusting osd_memory_target on smithi077 to 4288M 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:51.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:51.195 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:51.195 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:51 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:55:51.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:55:51.301+0000 7fcbfe7fc700 1 -- 172.21.15.104:0/2089412580 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcbf8012410 con 0x7fcc000fbe40 2024-02-19T18:55:51.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67533]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:55:51.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67533]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:51.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67533]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:51.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67533]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:51.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67565]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:55:51.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67565]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:51.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67565]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:51.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67565]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:51.991 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67620]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:55:51.992 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67620]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:51.992 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67620]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:51.992 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67620]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:52.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67645]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:55:52.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67645]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:52.250 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67645]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:52.250 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:51 smithi077 sudo[67645]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:52.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:52.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 ceph-mon[29247]: osdmap e23: 3 total, 3 up, 3 in 2024-02-19T18:55:52.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 ceph-mon[29247]: mgrmap e19: smithi077.yiiqpd(active, since 2m), standbys: smithi104.sdpzmi 2024-02-19T18:55:52.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/481090085' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15c9aad-95fd-4446-b3da-4f447fc3f7a1"}]: dispatch 2024-02-19T18:55:52.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 ceph-mon[29247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15c9aad-95fd-4446-b3da-4f447fc3f7a1"}]: dispatch 2024-02-19T18:55:52.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 ceph-mon[29247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d15c9aad-95fd-4446-b3da-4f447fc3f7a1"}]': finished 2024-02-19T18:55:52.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 ceph-mon[29247]: osdmap e24: 4 total, 3 up, 4 in 2024-02-19T18:55:52.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:55:52.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67651]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:55:52.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67651]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:52.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67651]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:52.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67651]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:52.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:52.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 ceph-mon[42320]: osdmap e23: 3 total, 3 up, 3 in 2024-02-19T18:55:52.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 ceph-mon[42320]: mgrmap e19: smithi077.yiiqpd(active, since 2m), standbys: smithi104.sdpzmi 2024-02-19T18:55:52.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 ceph-mon[42320]: from='client.? 172.21.15.104:0/481090085' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15c9aad-95fd-4446-b3da-4f447fc3f7a1"}]: dispatch 2024-02-19T18:55:52.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 ceph-mon[42320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d15c9aad-95fd-4446-b3da-4f447fc3f7a1"}]: dispatch 2024-02-19T18:55:52.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 ceph-mon[42320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d15c9aad-95fd-4446-b3da-4f447fc3f7a1"}]': finished 2024-02-19T18:55:52.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 ceph-mon[42320]: osdmap e24: 4 total, 3 up, 4 in 2024-02-19T18:55:52.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:55:52.763 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67698]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:55:52.763 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67698]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:52.763 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67698]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:52.763 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67698]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:53.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67732]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:55:53.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:53.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67732]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:53.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:52 smithi077 sudo[67732]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:53.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 sudo[45747]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-19T18:55:53.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 sudo[45747]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:53.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:52 smithi104 sudo[45747]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:53.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 sudo[45747]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:53 smithi077 ceph-mon[29247]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:53 smithi077 ceph-mon[29247]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:55:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:53 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/1535488487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:55:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:53 smithi077 ceph-mon[29247]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:55:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:53 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:55:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:53 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:55:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:53 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:55:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:53 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 sudo[45752]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 sudo[45752]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 sudo[45752]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 sudo[45752]: pam_unix(sudo:session): session closed for user root 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 ceph-mon[42320]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 ceph-mon[42320]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 ceph-mon[42320]: from='client.? 172.21.15.104:0/1535488487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 ceph-mon[42320]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:55:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T18:55:53.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:53 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T18:55:54.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:54 smithi077 ceph-mon[29247]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:55:54.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:54 smithi077 ceph-mon[29247]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:55:54.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:54 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:54.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:54 smithi104 ceph-mon[42320]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-19T18:55:54.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:54 smithi104 ceph-mon[42320]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-19T18:55:54.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:54 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:55 smithi077 ceph-mon[29247]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:55 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:55 smithi104 ceph-mon[42320]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:55:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:55:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:55 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:55:57.551 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:57 smithi104 ceph-mon[42320]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:57.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:57 smithi077 ceph-mon[29247]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:55:59.132 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:58 smithi104 ceph-mon[42320]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-19T18:55:59.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:58 smithi077 ceph-mon[29247]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-02-19T18:56:00.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:59 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T18:56:00.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:59 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:00.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:55:59 smithi077 ceph-mon[29247]: Deploying daemon osd.3 on smithi104 2024-02-19T18:56:00.311 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:59 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-19T18:56:00.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:59 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:00.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:55:59 smithi104 ceph-mon[42320]: Deploying daemon osd.3 on smithi104 2024-02-19T18:56:01.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:00 smithi104 ceph-mon[42320]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-19T18:56:01.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:00 smithi077 ceph-mon[29247]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-02-19T18:56:03.281 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:03 smithi104 ceph-mon[42320]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-19T18:56:03.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:03 smithi077 ceph-mon[29247]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-02-19T18:56:05.493 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:05 smithi104 ceph-mon[42320]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-19T18:56:05.493 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:05 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:05.493 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:05 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:05.493 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:05 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:05.493 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:05 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:05.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:05 smithi077 ceph-mon[29247]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-02-19T18:56:05.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:05.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:05.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:05.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:05 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:07.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:07 smithi104 ceph-mon[42320]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:56:07.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:07 smithi077 ceph-mon[29247]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:56:07.663 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 3 on host 'smithi104' 2024-02-19T18:56:07.663 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.645+0000 7fcbfe7fc700 1 -- 172.21.15.104:0/2089412580 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcc00066660 con 0x7fcbe8069210 2024-02-19T18:56:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcbe8069210 msgr2=0x7fcbe806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcbe8069210 0x7fcbe806b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009fd0 tx=0x7fcbf400b040 comp rx=0 tx=0).stop 2024-02-19T18:56:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 msgr2=0x7fcc000fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 0x7fcc000fff00 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8015040 tx=0x7fcbf8004d10 comp rx=0 tx=0).stop 2024-02-19T18:56:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 shutdown_connections 2024-02-19T18:56:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcbe8069210 0x7fcbe806b6c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcc000fbe40 0x7fcc000fff00 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:07.665 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 --2- 172.21.15.104:0/2089412580 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcc000fcfb0 0x7fcc000fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 >> 172.21.15.104:0/2089412580 conn(0x7fcc000f7520 msgr2=0x7fcc00065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 shutdown_connections 2024-02-19T18:56:07.666 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:07.648+0000 7fcc07ff0700 1 -- 172.21.15.104:0/2089412580 wait complete. 2024-02-19T18:56:08.238 DEBUG:teuthology.orchestra.run.smithi104:osd.3> sudo journalctl -f -n 0 -u ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.3.service 2024-02-19T18:56:08.242 INFO:tasks.cephadm:Deploying osd.4 on smithi104 with /dev/vg_nvme/lv_3... 2024-02-19T18:56:08.242 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-19T18:56:08.544 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi104/config 2024-02-19T18:56:08.875 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:08 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:08.875 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:08 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:08.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:08 smithi104 ceph-mon[42320]: from='osd.3 [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:56:08.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:08 smithi104 ceph-mon[42320]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:56:08.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:08 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:08.876 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:08 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:09.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:08 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:09.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:08 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:09.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:08 smithi077 ceph-mon[29247]: from='osd.3 [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:56:09.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:08 smithi077 ceph-mon[29247]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-19T18:56:09.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:08 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:09.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:08 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:09.869 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:09 smithi104 ceph-mon[42320]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:56:09.870 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:09 smithi104 ceph-mon[42320]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T18:56:09.870 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:09 smithi104 ceph-mon[42320]: osdmap e25: 4 total, 3 up, 4 in 2024-02-19T18:56:09.870 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:09 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:56:09.870 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:09 smithi104 ceph-mon[42320]: from='osd.3 [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:56:09.870 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:09 smithi104 ceph-mon[42320]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:56:10.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:09 smithi077 ceph-mon[29247]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-02-19T18:56:10.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:09 smithi077 ceph-mon[29247]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-19T18:56:10.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:09 smithi077 ceph-mon[29247]: osdmap e25: 4 total, 3 up, 4 in 2024-02-19T18:56:10.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:09 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:56:10.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:09 smithi077 ceph-mon[29247]: from='osd.3 [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:56:10.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:09 smithi077 ceph-mon[29247]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:56:10.312 INFO:journalctl@ceph.osd.3.smithi104.stdout:Feb 19 18:56:09 smithi104 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-3[47771]: 2024-02-19T18:56:09.852+0000 7f2f0f130700 -1 osd.3 0 waiting for initial osdmap 2024-02-19T18:56:10.312 INFO:journalctl@ceph.osd.3.smithi104.stdout:Feb 19 18:56:09 smithi104 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-3[47771]: 2024-02-19T18:56:09.860+0000 7f2f09f2b700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:56:10.948 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:10 smithi104 ceph-mon[42320]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-02-19T18:56:10.948 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:10 smithi104 ceph-mon[42320]: osdmap e26: 4 total, 3 up, 4 in 2024-02-19T18:56:10.948 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:10 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:56:10.948 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:10 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:56:10.948 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:10 smithi104 ceph-mon[42320]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:56:11.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:10 smithi077 ceph-mon[29247]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-02-19T18:56:11.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:10 smithi077 ceph-mon[29247]: osdmap e26: 4 total, 3 up, 4 in 2024-02-19T18:56:11.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:10 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:56:11.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:10 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:56:11.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:10 smithi077 ceph-mon[29247]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-02-19T18:56:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-19T18:56:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-19T18:56:11.519 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-02-19T18:56:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-02-19T18:56:11.519 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132406 s, 792 MB/s 2024-02-19T18:56:11.519 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-02-19T18:56:12.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:11 smithi104 ceph-mon[42320]: purged_snaps scrub starts 2024-02-19T18:56:12.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:11 smithi104 ceph-mon[42320]: purged_snaps scrub ok 2024-02-19T18:56:12.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:11 smithi104 ceph-mon[42320]: osd.3 [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] boot 2024-02-19T18:56:12.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:11 smithi104 ceph-mon[42320]: osdmap e27: 4 total, 4 up, 4 in 2024-02-19T18:56:12.178 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:11 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:56:12.202 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-02-19T18:56:12.302 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:11 smithi077 ceph-mon[29247]: purged_snaps scrub starts 2024-02-19T18:56:12.302 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:11 smithi077 ceph-mon[29247]: purged_snaps scrub ok 2024-02-19T18:56:12.302 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:11 smithi077 ceph-mon[29247]: osd.3 [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] boot 2024-02-19T18:56:12.302 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:11 smithi077 ceph-mon[29247]: osdmap e27: 4 total, 4 up, 4 in 2024-02-19T18:56:12.302 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:11 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T18:56:12.453 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi104/config 2024-02-19T18:56:13.134 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: osdmap e28: 4 total, 4 up, 4 in 2024-02-19T18:56:13.134 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:13.134 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:13.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:13.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:56:13.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:13.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:13.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:13.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:13.135 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:12 smithi104 ceph-mon[42320]: osdmap e29: 4 total, 4 up, 4 in 2024-02-19T18:56:13.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: osdmap e28: 4 total, 4 up, 4 in 2024-02-19T18:56:13.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:13.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:13.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:13.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:56:13.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:13.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:13.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:13.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:13.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:12 smithi077 ceph-mon[29247]: osdmap e29: 4 total, 4 up, 4 in 2024-02-19T18:56:14.108 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:13 smithi104 ceph-mon[42320]: Detected new or changed devices on smithi104 2024-02-19T18:56:14.108 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:13 smithi104 ceph-mon[42320]: Adjusting osd_memory_target on smithi104 to 15936M 2024-02-19T18:56:14.108 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:13 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:14.108 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:13 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:14.109 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:13 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:14.109 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:13 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:14.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:13 smithi077 ceph-mon[29247]: Detected new or changed devices on smithi104 2024-02-19T18:56:14.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:13 smithi077 ceph-mon[29247]: Adjusting osd_memory_target on smithi104 to 15936M 2024-02-19T18:56:14.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:13 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:14.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:13 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:14.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:13 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:14.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:13 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.296+0000 7fd235528700 1 -- 172.21.15.104:0/2798782787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fb220 msgr2=0x7fd2300fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.296+0000 7fd235528700 1 --2- 172.21.15.104:0/2798782787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fb220 0x7fd2300fb630 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd220009ce0 tx=0x7fd220004d10 comp rx=0 tx=0).stop 2024-02-19T18:56:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.296+0000 7fd235528700 1 -- 172.21.15.104:0/2798782787 shutdown_connections 2024-02-19T18:56:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.296+0000 7fd235528700 1 --2- 172.21.15.104:0/2798782787 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2300fe2b0 0x7fd2300fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.296+0000 7fd235528700 1 --2- 172.21.15.104:0/2798782787 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fb220 0x7fd2300fb630 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.296+0000 7fd235528700 1 -- 172.21.15.104:0/2798782787 >> 172.21.15.104:0/2798782787 conn(0x7fd2300f6ca0 msgr2=0x7fd2300f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.296+0000 7fd235528700 1 -- 172.21.15.104:0/2798782787 shutdown_connections 2024-02-19T18:56:14.314 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.296+0000 7fd235528700 1 -- 172.21.15.104:0/2798782787 wait complete. 2024-02-19T18:56:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd235528700 1 Processor -- start 2024-02-19T18:56:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd235528700 1 -- start start 2024-02-19T18:56:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd235528700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2300fb220 0x7fd23010da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd235528700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fe2b0 0x7fd23010df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd235528700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23010e550 con 0x7fd2300fb220 2024-02-19T18:56:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd235528700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23010edf0 con 0x7fd2300fe2b0 2024-02-19T18:56:14.316 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd22e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fe2b0 0x7fd23010df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd22e7fc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fe2b0 0x7fd23010df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:38456/0 (socket says 172.21.15.104:38456) 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd22e7fc700 1 -- 172.21.15.104:0/2605734134 learned_addr learned my addr 172.21.15.104:0/2605734134 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd22effd700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2300fb220 0x7fd23010da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd22e7fc700 1 -- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2300fb220 msgr2=0x7fd23010da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd22e7fc700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2300fb220 0x7fd23010da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd22e7fc700 1 -- 172.21.15.104:0/2605734134 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd220009940 con 0x7fd2300fe2b0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.297+0000 7fd22effd700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2300fb220 0x7fd23010da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.298+0000 7fd22e7fc700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fe2b0 0x7fd23010df50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd21800eb40 tx=0x7fd21800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.298+0000 7fd2277fe700 1 -- 172.21.15.104:0/2605734134 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd218007cc0 con 0x7fd2300fe2b0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.298+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23010f070 con 0x7fd2300fe2b0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.298+0000 7fd2277fe700 1 -- 172.21.15.104:0/2605734134 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd21800f040 con 0x7fd2300fe2b0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.298+0000 7fd2277fe700 1 -- 172.21.15.104:0/2605734134 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd21800ca10 con 0x7fd2300fe2b0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.298+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23010f5b0 con 0x7fd2300fe2b0 2024-02-19T18:56:14.317 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.299+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23004bed0 con 0x7fd2300fe2b0 2024-02-19T18:56:14.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.299+0000 7fd2277fe700 1 -- 172.21.15.104:0/2605734134 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd21800cc80 con 0x7fd2300fe2b0 2024-02-19T18:56:14.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.299+0000 7fd2277fe700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd21c0692a0 0x7fd21c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:14.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.300+0000 7fd2277fe700 1 -- 172.21.15.104:0/2605734134 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fd218086740 con 0x7fd2300fe2b0 2024-02-19T18:56:14.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.300+0000 7fd22effd700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd21c0692a0 0x7fd21c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:14.318 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.300+0000 7fd22effd700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd21c0692a0 0x7fd21c06b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd220015040 tx=0x7fd220004d10 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:14.321 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.304+0000 7fd2277fe700 1 -- 172.21.15.104:0/2605734134 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd21804dbd0 con 0x7fd2300fe2b0 2024-02-19T18:56:14.511 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:14.493+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fd23010fa00 con 0x7fd21c0692a0 2024-02-19T18:56:15.143 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:14 smithi104 ceph-mon[42320]: osdmap e30: 4 total, 4 up, 4 in 2024-02-19T18:56:15.143 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:14 smithi104 ceph-mon[42320]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:15.143 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:14 smithi104 ceph-mon[42320]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:15.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:14 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:56:15.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:14 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:56:15.144 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:14 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:15.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:14 smithi077 ceph-mon[29247]: osdmap e30: 4 total, 4 up, 4 in 2024-02-19T18:56:15.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:14 smithi077 ceph-mon[29247]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:15.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:14 smithi077 ceph-mon[29247]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:15.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:14 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:56:15.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:14 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:56:15.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:14 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:15.989 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:15 smithi104 ceph-mon[42320]: osdmap e31: 4 total, 4 up, 4 in 2024-02-19T18:56:16.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:15 smithi077 ceph-mon[29247]: osdmap e31: 4 total, 4 up, 4 in 2024-02-19T18:56:17.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:16 smithi077 ceph-mon[29247]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:17.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:16 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3342127287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a37971a1-f37c-4136-ae69-8a9999870db4"}]: dispatch 2024-02-19T18:56:17.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:16 smithi077 ceph-mon[29247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a37971a1-f37c-4136-ae69-8a9999870db4"}]: dispatch 2024-02-19T18:56:17.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:16 smithi077 ceph-mon[29247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a37971a1-f37c-4136-ae69-8a9999870db4"}]': finished 2024-02-19T18:56:17.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:16 smithi077 ceph-mon[29247]: osdmap e32: 5 total, 4 up, 5 in 2024-02-19T18:56:17.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:16 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:17.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:16 smithi104 ceph-mon[42320]: pgmap v100: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:17.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:16 smithi104 ceph-mon[42320]: from='client.? 172.21.15.104:0/3342127287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a37971a1-f37c-4136-ae69-8a9999870db4"}]: dispatch 2024-02-19T18:56:17.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:16 smithi104 ceph-mon[42320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a37971a1-f37c-4136-ae69-8a9999870db4"}]: dispatch 2024-02-19T18:56:17.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:16 smithi104 ceph-mon[42320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a37971a1-f37c-4136-ae69-8a9999870db4"}]': finished 2024-02-19T18:56:17.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:16 smithi104 ceph-mon[42320]: osdmap e32: 5 total, 4 up, 5 in 2024-02-19T18:56:17.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:16 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:18.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:17 smithi104 ceph-mon[42320]: from='client.? 172.21.15.104:0/2541760133' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:56:18.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:17 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/2541760133' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:56:19.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:18 smithi077 ceph-mon[29247]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-02-19T18:56:19.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:18 smithi104 ceph-mon[42320]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 109 KiB/s, 0 objects/s recovering 2024-02-19T18:56:20.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:20.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:21.270 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:20 smithi104 ceph-mon[42320]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-19T18:56:21.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:20 smithi077 ceph-mon[29247]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-02-19T18:56:23.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:23 smithi104 ceph-mon[42320]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-19T18:56:23.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:23 smithi077 ceph-mon[29247]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-19T18:56:24.528 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T18:56:24.528 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:24.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-19T18:56:24.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:25.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:25 smithi104 ceph-mon[42320]: Deploying daemon osd.4 on smithi104 2024-02-19T18:56:25.532 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:25 smithi104 ceph-mon[42320]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-19T18:56:25.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:25 smithi077 ceph-mon[29247]: Deploying daemon osd.4 on smithi104 2024-02-19T18:56:25.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:25 smithi077 ceph-mon[29247]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-02-19T18:56:27.449 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:27 smithi104 ceph-mon[42320]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-19T18:56:27.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:27 smithi077 ceph-mon[29247]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-19T18:56:29.117 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:28 smithi104 ceph-mon[42320]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-02-19T18:56:29.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:28 smithi077 ceph-mon[29247]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-02-19T18:56:30.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:30.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:30.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:30.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:30.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:30.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:31.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:31 smithi077 ceph-mon[29247]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:31.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:31 smithi104 ceph-mon[42320]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:33.078 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.059+0000 7fd2277fe700 1 -- 172.21.15.104:0/2605734134 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd23010fa00 con 0x7fd21c0692a0 2024-02-19T18:56:33.078 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 4 on host 'smithi104' 2024-02-19T18:56:33.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd21c0692a0 msgr2=0x7fd21c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:33.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd21c0692a0 0x7fd21c06b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fd220015040 tx=0x7fd220004d10 comp rx=0 tx=0).stop 2024-02-19T18:56:33.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fe2b0 msgr2=0x7fd23010df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:33.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fe2b0 0x7fd23010df50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd21800eb40 tx=0x7fd21800b040 comp rx=0 tx=0).stop 2024-02-19T18:56:33.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 shutdown_connections 2024-02-19T18:56:33.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd21c0692a0 0x7fd21c06b750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:33.080 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd2300fb220 0x7fd23010da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:33.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 --2- 172.21.15.104:0/2605734134 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd2300fe2b0 0x7fd23010df50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:33.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 >> 172.21.15.104:0/2605734134 conn(0x7fd2300f6ca0 msgr2=0x7fd2301014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:33.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 shutdown_connections 2024-02-19T18:56:33.081 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:33.061+0000 7fd235528700 1 -- 172.21.15.104:0/2605734134 wait complete. 2024-02-19T18:56:33.400 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:33 smithi104 ceph-mon[42320]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:33.401 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:33.401 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:33.401 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:33 smithi104 ceph-mon[42320]: from='osd.4 [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T18:56:33.401 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:33 smithi104 ceph-mon[42320]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T18:56:33.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:33 smithi077 ceph-mon[29247]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:33.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:33.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:33.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:33 smithi077 ceph-mon[29247]: from='osd.4 [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T18:56:33.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:33 smithi077 ceph-mon[29247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-19T18:56:33.723 DEBUG:teuthology.orchestra.run.smithi104:osd.4> sudo journalctl -f -n 0 -u ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.4.service 2024-02-19T18:56:33.726 INFO:tasks.cephadm:Deploying osd.5 on smithi104 with /dev/vg_nvme/lv_2... 2024-02-19T18:56:33.726 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-19T18:56:34.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:34.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:34.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:34 smithi077 ceph-mon[29247]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T18:56:34.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:34 smithi077 ceph-mon[29247]: osdmap e33: 5 total, 4 up, 5 in 2024-02-19T18:56:34.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:34.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:34 smithi077 ceph-mon[29247]: from='osd.4 [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:56:34.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:34 smithi077 ceph-mon[29247]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:56:34.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:34.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:34.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:34.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:34 smithi104 ceph-mon[42320]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-19T18:56:34.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:34 smithi104 ceph-mon[42320]: osdmap e33: 5 total, 4 up, 5 in 2024-02-19T18:56:34.796 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:34.797 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:34 smithi104 ceph-mon[42320]: from='osd.4 [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:56:34.797 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:34 smithi104 ceph-mon[42320]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:56:34.797 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:34.876 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi104/config 2024-02-19T18:56:35.312 INFO:journalctl@ceph.osd.4.smithi104.stdout:Feb 19 18:56:35 smithi104 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-4[54529]: 2024-02-19T18:56:35.032+0000 7f3fb11e1700 -1 osd.4 0 waiting for initial osdmap 2024-02-19T18:56:35.313 INFO:journalctl@ceph.osd.4.smithi104.stdout:Feb 19 18:56:35 smithi104 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-4[54529]: 2024-02-19T18:56:35.040+0000 7f3fabfdc700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:56:35.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:35 smithi104 ceph-mon[42320]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:35.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:35 smithi104 ceph-mon[42320]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-02-19T18:56:35.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:35 smithi104 ceph-mon[42320]: osdmap e34: 5 total, 4 up, 5 in 2024-02-19T18:56:35.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:35 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:35.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:35 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:35.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:35 smithi077 ceph-mon[29247]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-02-19T18:56:35.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:35 smithi077 ceph-mon[29247]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-02-19T18:56:35.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:35 smithi077 ceph-mon[29247]: osdmap e34: 5 total, 4 up, 5 in 2024-02-19T18:56:35.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:35 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:35.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:35 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:37.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: purged_snaps scrub starts 2024-02-19T18:56:37.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: purged_snaps scrub ok 2024-02-19T18:56:37.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: osd.4 [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] boot 2024-02-19T18:56:37.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: osdmap e35: 5 total, 5 up, 5 in 2024-02-19T18:56:37.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: Detected new or changed devices on smithi104 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: Adjusting osd_memory_target on smithi104 to 7968M 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:37.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:37 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: purged_snaps scrub starts 2024-02-19T18:56:37.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: purged_snaps scrub ok 2024-02-19T18:56:37.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: osd.4 [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] boot 2024-02-19T18:56:37.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: osdmap e35: 5 total, 5 up, 5 in 2024-02-19T18:56:37.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T18:56:37.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: Detected new or changed devices on smithi104 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: Adjusting osd_memory_target on smithi104 to 7968M 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:56:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:37 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-19T18:56:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-19T18:56:37.908 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10+0 records in 2024-02-19T18:56:37.908 INFO:teuthology.orchestra.run.smithi104.stderr:10+0 records out 2024-02-19T18:56:37.909 INFO:teuthology.orchestra.run.smithi104.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140172 s, 748 MB/s 2024-02-19T18:56:37.909 INFO:teuthology.orchestra.run.smithi104.stderr:--> Zapping successful for: 2024-02-19T18:56:38.283 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:38 smithi104 ceph-mon[42320]: osdmap e36: 5 total, 5 up, 5 in 2024-02-19T18:56:38.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:38 smithi077 ceph-mon[29247]: osdmap e36: 5 total, 5 up, 5 in 2024-02-19T18:56:38.710 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-02-19T18:56:38.962 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi104/config 2024-02-19T18:56:39.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:39 smithi077 ceph-mon[29247]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:39.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:39 smithi104 ceph-mon[42320]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:40.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.341+0000 7fa6455de700 1 -- 172.21.15.104:0/3237289743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 msgr2=0x7fa6400fc290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:40.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.341+0000 7fa6455de700 1 --2- 172.21.15.104:0/3237289743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 0x7fa6400fc290 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fa630004940 tx=0x7fa630009e30 comp rx=0 tx=0).stop 2024-02-19T18:56:40.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 -- 172.21.15.104:0/3237289743 shutdown_connections 2024-02-19T18:56:40.361 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 --2- 172.21.15.104:0/3237289743 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6400fd080 0x7fa6400fd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 --2- 172.21.15.104:0/3237289743 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 0x7fa6400fc290 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 -- 172.21.15.104:0/3237289743 >> 172.21.15.104:0/3237289743 conn(0x7fa6400f7590 msgr2=0x7fa6400f99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 -- 172.21.15.104:0/3237289743 shutdown_connections 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 -- 172.21.15.104:0/3237289743 wait complete. 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 Processor -- start 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 -- start start 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 0x7fa64010c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6400fd080 0x7fa64010cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64010d470 con 0x7fa6400fd080 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.342+0000 7fa6455de700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64010dcb0 con 0x7fa6400fbe80 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa63effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 0x7fa64010c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa63effd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 0x7fa64010c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:48948/0 (socket says 172.21.15.104:48948) 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa63effd700 1 -- 172.21.15.104:0/1898929301 learned_addr learned my addr 172.21.15.104:0/1898929301 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:56:40.362 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa63effd700 1 -- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6400fd080 msgr2=0x7fa64010cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa63effd700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6400fd080 0x7fa64010cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa63effd700 1 -- 172.21.15.104:0/1898929301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa630005040 con 0x7fa6400fbe80 2024-02-19T18:56:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa63effd700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 0x7fa64010c8a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa630009380 tx=0x7fa630009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa6277fe700 1 -- 172.21.15.104:0/1898929301 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa63000e040 con 0x7fa6400fbe80 2024-02-19T18:56:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64010df30 con 0x7fa6400fbe80 2024-02-19T18:56:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa6277fe700 1 -- 172.21.15.104:0/1898929301 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa63000bcc0 con 0x7fa6400fbe80 2024-02-19T18:56:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa6277fe700 1 -- 172.21.15.104:0/1898929301 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa63001c630 con 0x7fa6400fbe80 2024-02-19T18:56:40.363 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.343+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa64010e470 con 0x7fa6400fbe80 2024-02-19T18:56:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.344+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64004bed0 con 0x7fa6400fbe80 2024-02-19T18:56:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.345+0000 7fa6277fe700 1 -- 172.21.15.104:0/1898929301 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa630013420 con 0x7fa6400fbe80 2024-02-19T18:56:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.345+0000 7fa6277fe700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa6280692a0 0x7fa62806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:56:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.345+0000 7fa6277fe700 1 -- 172.21.15.104:0/1898929301 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fa6300880b0 con 0x7fa6400fbe80 2024-02-19T18:56:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.345+0000 7fa63e7fc700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa6280692a0 0x7fa62806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:56:40.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.346+0000 7fa63e7fc700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa6280692a0 0x7fa62806b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa634006fd0 tx=0x7fa634006db0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:56:40.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.349+0000 7fa6277fe700 1 -- 172.21.15.104:0/1898929301 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa630058360 con 0x7fa6400fbe80 2024-02-19T18:56:40.557 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:40.539+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa6401019d0 con 0x7fa6280692a0 2024-02-19T18:56:41.484 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:41 smithi104 ceph-mon[42320]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:41.485 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:41 smithi104 ceph-mon[42320]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:41.485 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:56:41.485 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:56:41.485 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:41.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:41 smithi077 ceph-mon[29247]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:41.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:41 smithi077 ceph-mon[29247]: from='client.24161 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:56:41.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-19T18:56:41.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-19T18:56:41.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:43.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:43 smithi077 ceph-mon[29247]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:43.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:43 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/305063816' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffc028a3-833b-4811-a38b-a47f1c6e15da"}]: dispatch 2024-02-19T18:56:43.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:43 smithi077 ceph-mon[29247]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffc028a3-833b-4811-a38b-a47f1c6e15da"}]: dispatch 2024-02-19T18:56:43.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:43 smithi077 ceph-mon[29247]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffc028a3-833b-4811-a38b-a47f1c6e15da"}]': finished 2024-02-19T18:56:43.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:43 smithi077 ceph-mon[29247]: osdmap e37: 6 total, 5 up, 6 in 2024-02-19T18:56:43.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:43 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:56:43.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:43 smithi104 ceph-mon[42320]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:43.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:43 smithi104 ceph-mon[42320]: from='client.? 172.21.15.104:0/305063816' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffc028a3-833b-4811-a38b-a47f1c6e15da"}]: dispatch 2024-02-19T18:56:43.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:43 smithi104 ceph-mon[42320]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ffc028a3-833b-4811-a38b-a47f1c6e15da"}]: dispatch 2024-02-19T18:56:43.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:43 smithi104 ceph-mon[42320]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ffc028a3-833b-4811-a38b-a47f1c6e15da"}]': finished 2024-02-19T18:56:43.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:43 smithi104 ceph-mon[42320]: osdmap e37: 6 total, 5 up, 6 in 2024-02-19T18:56:43.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:43 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:56:44.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:44 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/4255629083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:56:44.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:44 smithi104 ceph-mon[42320]: from='client.? 172.21.15.104:0/4255629083' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-19T18:56:45.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:45 smithi077 ceph-mon[29247]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:45.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:45 smithi104 ceph-mon[42320]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:47.528 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:47 smithi104 ceph-mon[42320]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:47.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:47 smithi077 ceph-mon[29247]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:49.149 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:48 smithi104 ceph-mon[42320]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:49.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:48 smithi077 ceph-mon[29247]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:50.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:50.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:56:51.055 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:50 smithi104 ceph-mon[42320]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:51.055 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:50 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T18:56:51.055 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:50 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:51.056 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:50 smithi104 ceph-mon[42320]: Deploying daemon osd.5 on smithi104 2024-02-19T18:56:51.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:50 smithi077 ceph-mon[29247]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:51.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:50 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-19T18:56:51.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:50 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:56:51.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:50 smithi077 ceph-mon[29247]: Deploying daemon osd.5 on smithi104 2024-02-19T18:56:53.258 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:53 smithi104 ceph-mon[42320]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:53 smithi077 ceph-mon[29247]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:55.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:55 smithi077 ceph-mon[29247]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:55 smithi104 ceph-mon[42320]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:56.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:56 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:56.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:56 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:56.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:56 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:56.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:56 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:56.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:56 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:56:56.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:56 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:56:57.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:57 smithi104 ceph-mon[42320]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:57.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:57 smithi077 ceph-mon[29247]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:59.271 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:56:58 smithi104 ceph-mon[42320]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:59.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:56:58 smithi077 ceph-mon[29247]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:56:59.322 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.305+0000 7fa6277fe700 1 -- 172.21.15.104:0/1898929301 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa6401019d0 con 0x7fa6280692a0 2024-02-19T18:56:59.323 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 5 on host 'smithi104' 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa6280692a0 msgr2=0x7fa62806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa6280692a0 0x7fa62806b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa634006fd0 tx=0x7fa634006db0 comp rx=0 tx=0).stop 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 msgr2=0x7fa64010c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 0x7fa64010c8a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fa630009380 tx=0x7fa630009e30 comp rx=0 tx=0).stop 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 shutdown_connections 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa6280692a0 0x7fa62806b750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6400fbe80 0x7fa64010c8a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 --2- 172.21.15.104:0/1898929301 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6400fd080 0x7fa64010cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:56:59.325 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 >> 172.21.15.104:0/1898929301 conn(0x7fa6400f7590 msgr2=0x7fa6401002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:56:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 shutdown_connections 2024-02-19T18:56:59.326 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:56:59.307+0000 7fa6455de700 1 -- 172.21.15.104:0/1898929301 wait complete. 2024-02-19T18:56:59.948 DEBUG:teuthology.orchestra.run.smithi104:osd.5> sudo journalctl -f -n 0 -u ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.5.service 2024-02-19T18:56:59.951 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-02-19T18:56:59.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-19T18:57:00.219 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:00.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:00 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:00.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:00 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:00.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:00 smithi077 ceph-mon[29247]: from='osd.5 [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T18:57:00.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:00 smithi077 ceph-mon[29247]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T18:57:00.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:00 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:00.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:00 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:00.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:00 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:00.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:00 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:00.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:00 smithi104 ceph-mon[42320]: from='osd.5 [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T18:57:00.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:00 smithi104 ceph-mon[42320]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-19T18:57:00.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:00 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:00.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:00 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:01.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:01 smithi077 ceph-mon[29247]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:57:01.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:01 smithi077 ceph-mon[29247]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T18:57:01.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:01 smithi077 ceph-mon[29247]: osdmap e38: 6 total, 5 up, 6 in 2024-02-19T18:57:01.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:01 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:57:01.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:01 smithi077 ceph-mon[29247]: from='osd.5 [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:57:01.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:01 smithi077 ceph-mon[29247]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:57:01.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:01 smithi104 ceph-mon[42320]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:57:01.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:01 smithi104 ceph-mon[42320]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-19T18:57:01.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:01 smithi104 ceph-mon[42320]: osdmap e38: 6 total, 5 up, 6 in 2024-02-19T18:57:01.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:01 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:57:01.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:01 smithi104 ceph-mon[42320]: from='osd.5 [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:57:01.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:01 smithi104 ceph-mon[42320]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-02-19T18:57:01.563 INFO:journalctl@ceph.osd.5.smithi104.stdout:Feb 19 18:57:01 smithi104 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-5[61281]: 2024-02-19T18:57:01.311+0000 7f9a8efd4700 -1 osd.5 0 waiting for initial osdmap 2024-02-19T18:57:01.563 INFO:journalctl@ceph.osd.5.smithi104.stdout:Feb 19 18:57:01 smithi104 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-5[61281]: 2024-02-19T18:57:01.320+0000 7f9a895ce700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.973+0000 7fb64b65a700 1 -- 172.21.15.77:0/1908857278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 msgr2=0x7fb6440fafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.973+0000 7fb64b65a700 1 --2- 172.21.15.77:0/1908857278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 0x7fb6440fafe0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb634004940 tx=0x7fb634009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.973+0000 7fb64b65a700 1 -- 172.21.15.77:0/1908857278 shutdown_connections 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.973+0000 7fb64b65a700 1 --2- 172.21.15.77:0/1908857278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6440fb5b0 0x7fb6441027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.973+0000 7fb64b65a700 1 --2- 172.21.15.77:0/1908857278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 0x7fb6440fafe0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.973+0000 7fb64b65a700 1 -- 172.21.15.77:0/1908857278 >> 172.21.15.77:0/1908857278 conn(0x7fb6440f6840 msgr2=0x7fb6440f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.973+0000 7fb64b65a700 1 -- 172.21.15.77:0/1908857278 shutdown_connections 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.973+0000 7fb64b65a700 1 -- 172.21.15.77:0/1908857278 wait complete. 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb64b65a700 1 Processor -- start 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb64b65a700 1 -- start start 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb64b65a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 0x7fb64410d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb64b65a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6440fb5b0 0x7fb64410ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:01.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb64b65a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64410e3f0 con 0x7fb6440fabd0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb64b65a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64410ec90 con 0x7fb6440fb5b0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb6493f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 0x7fb64410d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb6493f6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 0x7fb64410d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55428/0 (socket says 172.21.15.77:55428) 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb6493f6700 1 -- 172.21.15.77:0/3766113996 learned_addr learned my addr 172.21.15.77:0/3766113996 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb648bf5700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6440fb5b0 0x7fb64410ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb6493f6700 1 -- 172.21.15.77:0/3766113996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6440fb5b0 msgr2=0x7fb64410ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb6493f6700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6440fb5b0 0x7fb64410ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb6493f6700 1 -- 172.21.15.77:0/3766113996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb634005040 con 0x7fb6440fabd0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.974+0000 7fb6493f6700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 0x7fb64410d8b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb634000c00 tx=0x7fb634009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.975+0000 7fb639ffb700 1 -- 172.21.15.77:0/3766113996 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb634021070 con 0x7fb6440fabd0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.975+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64410ef10 con 0x7fb6440fabd0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.975+0000 7fb639ffb700 1 -- 172.21.15.77:0/3766113996 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb63401c400 con 0x7fb6440fabd0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.975+0000 7fb639ffb700 1 -- 172.21.15.77:0/3766113996 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63400e070 con 0x7fb6440fabd0 2024-02-19T18:57:01.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.975+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64410f3f0 con 0x7fb6440fabd0 2024-02-19T18:57:01.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.976+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb644107a40 con 0x7fb6440fabd0 2024-02-19T18:57:01.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.976+0000 7fb639ffb700 1 -- 172.21.15.77:0/3766113996 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb63401ca50 con 0x7fb6440fabd0 2024-02-19T18:57:01.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.977+0000 7fb639ffb700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb6300692a0 0x7fb63006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:01.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.977+0000 7fb639ffb700 1 -- 172.21.15.77:0/3766113996 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fb634087f40 con 0x7fb6440fabd0 2024-02-19T18:57:01.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.977+0000 7fb648bf5700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb6300692a0 0x7fb63006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:01.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.977+0000 7fb648bf5700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb6300692a0 0x7fb63006b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb640009b80 tx=0x7fb64000b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:01.998 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:01.981+0000 7fb639ffb700 1 -- 172.21.15.77:0/3766113996 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb6340b9bd0 con 0x7fb6440fabd0 2024-02-19T18:57:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.170+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb64404bed0 con 0x7fb6440fabd0 2024-02-19T18:57:02.188 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.170+0000 7fb639ffb700 1 -- 172.21.15.77:0/3766113996 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fb634058150 con 0x7fb6440fabd0 2024-02-19T18:57:02.188 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.172+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb6300692a0 msgr2=0x7fb63006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.172+0000 7fb64b65a700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb6300692a0 0x7fb63006b750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb640009b80 tx=0x7fb64000b040 comp rx=0 tx=0).stop 2024-02-19T18:57:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.172+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 msgr2=0x7fb64410d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.172+0000 7fb64b65a700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 0x7fb64410d8b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb634000c00 tx=0x7fb634009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.173+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 shutdown_connections 2024-02-19T18:57:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.173+0000 7fb64b65a700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb6300692a0 0x7fb63006b750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:02.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.173+0000 7fb64b65a700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb6440fabd0 0x7fb64410d8b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:02.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.173+0000 7fb64b65a700 1 --2- 172.21.15.77:0/3766113996 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb6440fb5b0 0x7fb64410ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:02.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.173+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 >> 172.21.15.77:0/3766113996 conn(0x7fb6440f6840 msgr2=0x7fb644101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:02.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.173+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 shutdown_connections 2024-02-19T18:57:02.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:02.173+0000 7fb64b65a700 1 -- 172.21.15.77:0/3766113996 wait complete. 2024-02-19T18:57:02.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:02 smithi104 ceph-mon[42320]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-02-19T18:57:02.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:02 smithi104 ceph-mon[42320]: osdmap e39: 6 total, 5 up, 6 in 2024-02-19T18:57:02.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:02 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:57:02.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:02 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3766113996' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T18:57:02.679 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:02 smithi077 ceph-mon[29247]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-02-19T18:57:02.679 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:02 smithi077 ceph-mon[29247]: osdmap e39: 6 total, 5 up, 6 in 2024-02-19T18:57:02.679 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:02 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:57:02.680 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:02 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3766113996' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T18:57:02.790 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1708368996,"num_in_osds":6,"osd_in_since":1708369002,"num_remapped_pgs":0} 2024-02-19T18:57:03.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:03 smithi077 ceph-mon[29247]: purged_snaps scrub starts 2024-02-19T18:57:03.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:03 smithi077 ceph-mon[29247]: purged_snaps scrub ok 2024-02-19T18:57:03.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:03 smithi077 ceph-mon[29247]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:57:03.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:03 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:57:03.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:03 smithi077 ceph-mon[29247]: osd.5 [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] boot 2024-02-19T18:57:03.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:03 smithi077 ceph-mon[29247]: osdmap e40: 6 total, 6 up, 6 in 2024-02-19T18:57:03.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:03 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:57:03.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:03 smithi104 ceph-mon[42320]: purged_snaps scrub starts 2024-02-19T18:57:03.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:03 smithi104 ceph-mon[42320]: purged_snaps scrub ok 2024-02-19T18:57:03.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:03 smithi104 ceph-mon[42320]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-02-19T18:57:03.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:03 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:57:03.686 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:03 smithi104 ceph-mon[42320]: osd.5 [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] boot 2024-02-19T18:57:03.687 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:03 smithi104 ceph-mon[42320]: osdmap e40: 6 total, 6 up, 6 in 2024-02-19T18:57:03.687 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:03 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T18:57:03.791 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd stat -f json 2024-02-19T18:57:04.059 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:04.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: osdmap e41: 6 total, 6 up, 6 in 2024-02-19T18:57:04.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:04.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:04.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:04.437 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:04.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:04.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:04.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:04.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:04.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: osdmap e41: 6 total, 6 up, 6 in 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:57:04.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:57:04.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:57:04.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:57:04.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:05.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:05 smithi077 ceph-mon[29247]: Detected new or changed devices on smithi104 2024-02-19T18:57:05.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:05 smithi077 ceph-mon[29247]: Adjusting osd_memory_target on smithi104 to 5312M 2024-02-19T18:57:05.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:05 smithi077 ceph-mon[29247]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:05.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.530+0000 7fa699ed2700 1 -- 172.21.15.77:0/3151645512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 msgr2=0x7fa6940fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.530+0000 7fa699ed2700 1 --2- 172.21.15.77:0/3151645512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 0x7fa6940fc250 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa684004940 tx=0x7fa684009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.530+0000 7fa699ed2700 1 -- 172.21.15.77:0/3151645512 shutdown_connections 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.530+0000 7fa699ed2700 1 --2- 172.21.15.77:0/3151645512 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6940fcfb0 0x7fa6940fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.530+0000 7fa699ed2700 1 --2- 172.21.15.77:0/3151645512 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 0x7fa6940fc250 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.530+0000 7fa699ed2700 1 -- 172.21.15.77:0/3151645512 >> 172.21.15.77:0/3151645512 conn(0x7fa6940f7520 msgr2=0x7fa6940f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.530+0000 7fa699ed2700 1 -- 172.21.15.77:0/3151645512 shutdown_connections 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.530+0000 7fa699ed2700 1 -- 172.21.15.77:0/3151645512 wait complete. 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa699ed2700 1 Processor -- start 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa699ed2700 1 -- start start 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa699ed2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 0x7fa69410d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa699ed2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6940fcfb0 0x7fa69410de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa699ed2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69410e480 con 0x7fa6940fbe40 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa699ed2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69410ed20 con 0x7fa6940fcfb0 2024-02-19T18:57:05.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa6937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 0x7fa69410d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa6937fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 0x7fa69410d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55444/0 (socket says 172.21.15.77:55444) 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa6937fe700 1 -- 172.21.15.77:0/1287701356 learned_addr learned my addr 172.21.15.77:0/1287701356 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa692ffd700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6940fcfb0 0x7fa69410de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa6937fe700 1 -- 172.21.15.77:0/1287701356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6940fcfb0 msgr2=0x7fa69410de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa6937fe700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6940fcfb0 0x7fa69410de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.531+0000 7fa6937fe700 1 -- 172.21.15.77:0/1287701356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684005040 con 0x7fa6940fbe40 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.532+0000 7fa6937fe700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 0x7fa69410d940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa6840093b0 tx=0x7fa684009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.532+0000 7fa67bfff700 1 -- 172.21.15.77:0/1287701356 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa68400e040 con 0x7fa6940fbe40 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.532+0000 7fa67bfff700 1 -- 172.21.15.77:0/1287701356 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa68400bcc0 con 0x7fa6940fbe40 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.532+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69410efa0 con 0x7fa6940fbe40 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.532+0000 7fa67bfff700 1 -- 172.21.15.77:0/1287701356 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa68401c630 con 0x7fa6940fbe40 2024-02-19T18:57:05.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.532+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69410f4e0 con 0x7fa6940fbe40 2024-02-19T18:57:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.533+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69404bed0 con 0x7fa6940fbe40 2024-02-19T18:57:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.533+0000 7fa67bfff700 1 -- 172.21.15.77:0/1287701356 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa6840139f0 con 0x7fa6940fbe40 2024-02-19T18:57:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.533+0000 7fa67bfff700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa67c0692a0 0x7fa67c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.534+0000 7fa692ffd700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa67c0692a0 0x7fa67c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.534+0000 7fa67bfff700 1 -- 172.21.15.77:0/1287701356 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa684088250 con 0x7fa6940fbe40 2024-02-19T18:57:05.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.534+0000 7fa692ffd700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa67c0692a0 0x7fa67c06b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa688007fd0 tx=0x7fa688009040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:05.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.538+0000 7fa67bfff700 1 -- 172.21.15.77:0/1287701356 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa684058350 con 0x7fa6940fbe40 2024-02-19T18:57:05.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.719+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fa69410f130 con 0x7fa6940fbe40 2024-02-19T18:57:05.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.719+0000 7fa67bfff700 1 -- 172.21.15.77:0/1287701356 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7fa69410f130 con 0x7fa6940fbe40 2024-02-19T18:57:05.737 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:05.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa67c0692a0 msgr2=0x7fa67c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:05.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa67c0692a0 0x7fa67c06b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa688007fd0 tx=0x7fa688009040 comp rx=0 tx=0).stop 2024-02-19T18:57:05.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 msgr2=0x7fa69410d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:05.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 0x7fa69410d940 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa6840093b0 tx=0x7fa684009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:05.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 shutdown_connections 2024-02-19T18:57:05.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa67c0692a0 0x7fa67c06b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:05.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa6940fbe40 0x7fa69410d940 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:05.739 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 --2- 172.21.15.77:0/1287701356 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa6940fcfb0 0x7fa69410de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:05.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 >> 172.21.15.77:0/1287701356 conn(0x7fa6940f7520 msgr2=0x7fa694065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:05.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 shutdown_connections 2024-02-19T18:57:05.740 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:05.721+0000 7fa699ed2700 1 -- 172.21.15.77:0/1287701356 wait complete. 2024-02-19T18:57:05.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:05 smithi104 ceph-mon[42320]: Detected new or changed devices on smithi104 2024-02-19T18:57:05.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:05 smithi104 ceph-mon[42320]: Adjusting osd_memory_target on smithi104 to 5312M 2024-02-19T18:57:05.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:05 smithi104 ceph-mon[42320]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:06.384 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1708369022,"num_in_osds":6,"osd_in_since":1708369002,"num_remapped_pgs":0} 2024-02-19T18:57:06.385 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T18:57:06.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:06 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1287701356' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T18:57:06.642 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:06.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:06 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1287701356' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-19T18:57:07.454 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:07 smithi077 ceph-mon[29247]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:07.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:07 smithi104 ceph-mon[42320]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.211+0000 7eff15094700 1 -- 172.21.15.77:0/1064349818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff10101e40 msgr2=0x7eff10102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.211+0000 7eff15094700 1 --2- 172.21.15.77:0/1064349818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff10101e40 0x7eff10102210 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7eff04004940 tx=0x7eff04009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.211+0000 7eff15094700 1 -- 172.21.15.77:0/1064349818 shutdown_connections 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.211+0000 7eff15094700 1 --2- 172.21.15.77:0/1064349818 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff101027e0 0x7eff100fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.211+0000 7eff15094700 1 --2- 172.21.15.77:0/1064349818 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff10101e40 0x7eff10102210 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.211+0000 7eff15094700 1 -- 172.21.15.77:0/1064349818 >> 172.21.15.77:0/1064349818 conn(0x7eff10071d90 msgr2=0x7eff10072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.211+0000 7eff15094700 1 -- 172.21.15.77:0/1064349818 shutdown_connections 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.211+0000 7eff15094700 1 -- 172.21.15.77:0/1064349818 wait complete. 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff15094700 1 Processor -- start 2024-02-19T18:57:08.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff15094700 1 -- start start 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff15094700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff10101e40 0x7eff1010eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff15094700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff101027e0 0x7eff1010f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff15094700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1010f5d0 con 0x7eff101027e0 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff15094700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1010f710 con 0x7eff10101e40 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff10101e40 0x7eff1010eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff101027e0 0x7eff1010f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0e59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff101027e0 0x7eff1010f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38828/0 (socket says 172.21.15.77:38828) 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0ed9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff10101e40 0x7eff1010eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.77:46182/0 (socket says 172.21.15.77:46182) 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0e59c700 1 -- 172.21.15.77:0/1099874376 learned_addr learned my addr 172.21.15.77:0/1099874376 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0e59c700 1 -- 172.21.15.77:0/1099874376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff10101e40 msgr2=0x7eff1010eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0e59c700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff10101e40 0x7eff1010eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0e59c700 1 -- 172.21.15.77:0/1099874376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff04005040 con 0x7eff101027e0 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.212+0000 7eff0e59c700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff101027e0 0x7eff1010f040 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7eff0000c7e0 tx=0x7eff00007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.213+0000 7efef77fe700 1 -- 172.21.15.77:0/1099874376 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff00012810 con 0x7eff101027e0 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.213+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff10117a70 con 0x7eff101027e0 2024-02-19T18:57:08.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.213+0000 7efef77fe700 1 -- 172.21.15.77:0/1099874376 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff00012e40 con 0x7eff101027e0 2024-02-19T18:57:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.213+0000 7efef77fe700 1 -- 172.21.15.77:0/1099874376 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff000102e0 con 0x7eff101027e0 2024-02-19T18:57:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.213+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff10117f80 con 0x7eff101027e0 2024-02-19T18:57:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.214+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff100ff150 con 0x7eff101027e0 2024-02-19T18:57:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.214+0000 7efef77fe700 1 -- 172.21.15.77:0/1099874376 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7eff00011380 con 0x7eff101027e0 2024-02-19T18:57:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.214+0000 7efef77fe700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7efef80692a0 0x7efef806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.215+0000 7efef77fe700 1 -- 172.21.15.77:0/1099874376 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7eff000871e0 con 0x7eff101027e0 2024-02-19T18:57:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.215+0000 7eff0ed9d700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7efef80692a0 0x7efef806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:08.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.215+0000 7eff0ed9d700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7efef80692a0 0x7efef806b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff04004f90 tx=0x7eff04009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:08.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.219+0000 7efef77fe700 1 -- 172.21.15.77:0/1099874376 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7eff000537d0 con 0x7eff101027e0 2024-02-19T18:57:08.419 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.402+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff10065ec0 con 0x7eff101027e0 2024-02-19T18:57:08.420 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.403+0000 7efef77fe700 1 -- 172.21.15.77:0/1099874376 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11072 (secure 0 0 0) 0x7eff00056df0 con 0x7eff101027e0 2024-02-19T18:57:08.421 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:08.421 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","created":"2024-02-19T18:49:39.682097+0000","modified":"2024-02-19T18:57:03.313927+0000","last_up_change":"2024-02-19T18:57:02.307357+0000","last_in_change":"2024-02-19T18:56:42.906075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:55:48.412324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fbc9e22d-2cb7-4421-b6d8-0dac7325974f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6803","nonce":806127290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6805","nonce":806127290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6809","nonce":806127290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6807","nonce":806127290}]},"public_addr":"172.21.15.77:6803/806127290","cluster_addr":"172.21.15.77:6805/806127290","heartbeat_back_addr":"172.21.15.77:6809/806127290","heartbeat_front_addr":"172.21.15.77:6807/806127290","state":["exists","up"]},{"osd":1,"uuid":"c544f75d-7ae5-4261-b55e-f2be62afb9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6811","nonce":3114250568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6813","nonce":3114250568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6817","nonce":3114250568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6815","nonce":3114250568}]},"public_addr":"172.21.15.77:6811/3114250568","cluster_addr":"172.21.15.77:6813/3114250568","heartbeat_back_addr":"172.21.15.77:6817/3114250568","heartbeat_front_addr":"172.21.15.77:6815/3114250568","state":["exists","up"]},{"osd":2,"uuid":"8463e27a-7421-42a1-8b87-7185b9f61766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6819","nonce":100949402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6821","nonce":100949402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6825","nonce":100949402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6823","nonce":100949402}]},"public_addr":"172.21.15.77:6819/100949402","cluster_addr":"172.21.15.77:6821/100949402","heartbeat_back_addr":"172.21.15.77:6825/100949402","heartbeat_front_addr":"172.21.15.77:6823/100949402","state":["exists","up"]},{"osd":3,"uuid":"d15c9aad-95fd-4446-b3da-4f447fc3f7a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6801","nonce":3308675957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6803","nonce":3308675957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6807","nonce":3308675957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6805","nonce":3308675957}]},"public_addr":"172.21.15.104:6801/3308675957","cluster_addr":"172.21.15.104:6803/3308675957","heartbeat_back_addr":"172.21.15.104:6807/3308675957","heartbeat_front_addr":"172.21.15.104:6805/3308675957","state":["exists","up"]},{"osd":4,"uuid":"a37971a1-f37c-4136-ae69-8a9999870db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6809","nonce":3596569551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6811","nonce":3596569551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6815","nonce":3596569551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6813","nonce":3596569551}]},"public_addr":"172.21.15.104:6809/3596569551","cluster_addr":"172.21.15.104:6811/3596569551","heartbeat_back_addr":"172.21.15.104:6815/3596569551","heartbeat_front_addr":"172.21.15.104:6813/3596569551","state":["exists","up"]},{"osd":5,"uuid":"ffc028a3-833b-4811-a38b-a47f1c6e15da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6817","nonce":2114623660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6819","nonce":2114623660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6823","nonce":2114623660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6821","nonce":2114623660}]},"public_addr":"172.21.15.104:6817/2114623660","cluster_addr":"172.21.15.104:6819/2114623660","heartbeat_back_addr":"172.21.15.104:6823/2114623660","heartbeat_front_addr":"172.21.15.104:6821/2114623660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:54:46.893755+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:55:15.359473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:55:45.118036+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:08.973267+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:34.107665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:00.315125+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2151174911":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/352143230":"2024-02-20T18:53:34.087006+0000","172.21.15.77:6800/3287530536":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/4032060278":"2024-02-20T18:53:34.087006+0000","172.21.15.77:6801/1028404574":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/2030253411":"2024-02-20T18:50:15.313914+0000","172.21.15.77:0/3240479911":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/2766229332":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/1552197174":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6800/1028404574":"2024-02-20T18:50:46.166039+0000","172.21.15.77:6801/2289192383":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6801/3287530536":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/1650327672":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/1405110191":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6800/2289192383":"2024-02-20T18:50:15.313914+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T18:57:08.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.404+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7efef80692a0 msgr2=0x7efef806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:08.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.404+0000 7eff15094700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7efef80692a0 0x7efef806b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7eff04004f90 tx=0x7eff04009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:08.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.404+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff101027e0 msgr2=0x7eff1010f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:08.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.404+0000 7eff15094700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff101027e0 0x7eff1010f040 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7eff0000c7e0 tx=0x7eff00007f90 comp rx=0 tx=0).stop 2024-02-19T18:57:08.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.405+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 shutdown_connections 2024-02-19T18:57:08.422 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.405+0000 7eff15094700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7efef80692a0 0x7efef806b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.405+0000 7eff15094700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff10101e40 0x7eff1010eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.405+0000 7eff15094700 1 --2- 172.21.15.77:0/1099874376 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff101027e0 0x7eff1010f040 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:08.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.405+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 >> 172.21.15.77:0/1099874376 conn(0x7eff10071d90 msgr2=0x7eff100fadf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:08.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.405+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 shutdown_connections 2024-02-19T18:57:08.423 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:08.405+0000 7eff15094700 1 -- 172.21.15.77:0/1099874376 wait complete. 2024-02-19T18:57:09.053 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:08 smithi077 ceph-mon[29247]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:09.053 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:08 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1099874376' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:57:09.078 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T18:55:48.412324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-02-19T18:57:09.078 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-19T18:57:09.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:08 smithi104 ceph-mon[42320]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:09.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:08 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1099874376' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:57:09.333 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:10.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.953+0000 7f8a8a259700 1 -- 172.21.15.77:0/1150585359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a84102f50 msgr2=0x7f8a84103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:10.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.953+0000 7f8a8a259700 1 --2- 172.21.15.77:0/1150585359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a84102f50 0x7f8a84103320 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a74004940 tx=0x7f8a74009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:10.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.953+0000 7f8a8a259700 1 -- 172.21.15.77:0/1150585359 shutdown_connections 2024-02-19T18:57:10.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.953+0000 7f8a8a259700 1 --2- 172.21.15.77:0/1150585359 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a841038f0 0x7f8a840fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:10.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.953+0000 7f8a8a259700 1 --2- 172.21.15.77:0/1150585359 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a84102f50 0x7f8a84103320 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:10.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.953+0000 7f8a8a259700 1 -- 172.21.15.77:0/1150585359 >> 172.21.15.77:0/1150585359 conn(0x7f8a840f8ae0 msgr2=0x7f8a840faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:10.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.953+0000 7f8a8a259700 1 -- 172.21.15.77:0/1150585359 shutdown_connections 2024-02-19T18:57:10.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.953+0000 7f8a8a259700 1 -- 172.21.15.77:0/1150585359 wait complete. 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a8a259700 1 Processor -- start 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a8a259700 1 -- start start 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a8a259700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a84102f50 0x7f8a84113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a8a259700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a841038f0 0x7f8a84114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a8a259700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a84114700 con 0x7f8a841038f0 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a8a259700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8410d9a0 con 0x7f8a84102f50 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a82ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a841038f0 0x7f8a84114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a82ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a841038f0 0x7f8a84114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38848/0 (socket says 172.21.15.77:38848) 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.954+0000 7f8a82ffd700 1 -- 172.21.15.77:0/2886281920 learned_addr learned my addr 172.21.15.77:0/2886281920 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a837fe700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a84102f50 0x7f8a84113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a82ffd700 1 -- 172.21.15.77:0/2886281920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a84102f50 msgr2=0x7f8a84113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a82ffd700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a84102f50 0x7f8a84113ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a82ffd700 1 -- 172.21.15.77:0/2886281920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a74005040 con 0x7f8a841038f0 2024-02-19T18:57:10.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a82ffd700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a841038f0 0x7f8a84114020 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800b790 tx=0x7f8a78009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:10.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a6bfff700 1 -- 172.21.15.77:0/2886281920 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a7800f6f0 con 0x7f8a841038f0 2024-02-19T18:57:10.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8410dc80 con 0x7f8a841038f0 2024-02-19T18:57:10.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a6bfff700 1 -- 172.21.15.77:0/2886281920 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a7800fd20 con 0x7f8a841038f0 2024-02-19T18:57:10.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a6bfff700 1 -- 172.21.15.77:0/2886281920 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a780103a0 con 0x7f8a841038f0 2024-02-19T18:57:10.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.955+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8410e1c0 con 0x7f8a841038f0 2024-02-19T18:57:10.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.956+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a840fc6e0 con 0x7f8a841038f0 2024-02-19T18:57:10.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.956+0000 7f8a6bfff700 1 -- 172.21.15.77:0/2886281920 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8a78011400 con 0x7f8a841038f0 2024-02-19T18:57:10.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.957+0000 7f8a6bfff700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8a6c0692b0 0x7f8a6c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:10.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.957+0000 7f8a837fe700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8a6c0692b0 0x7f8a6c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:10.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.957+0000 7f8a6bfff700 1 -- 172.21.15.77:0/2886281920 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8a78086d10 con 0x7f8a841038f0 2024-02-19T18:57:10.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.957+0000 7f8a837fe700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8a6c0692b0 0x7f8a6c06b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a7400b3b0 tx=0x7f8a74009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:10.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:10.961+0000 7f8a6bfff700 1 -- 172.21.15.77:0/2886281920 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8a78056d90 con 0x7f8a841038f0 2024-02-19T18:57:11.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.146+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a8404bed0 con 0x7f8a841038f0 2024-02-19T18:57:11.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.147+0000 7f8a6bfff700 1 -- 172.21.15.77:0/2886281920 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a78056920 con 0x7f8a841038f0 2024-02-19T18:57:11.165 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-02-19T18:57:11.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8a6c0692b0 msgr2=0x7f8a6c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:11.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8a6c0692b0 0x7f8a6c06b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a7400b3b0 tx=0x7f8a74009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a841038f0 msgr2=0x7f8a84114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a841038f0 0x7f8a84114020 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8a7800b790 tx=0x7f8a78009220 comp rx=0 tx=0).stop 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 shutdown_connections 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8a6c0692b0 0x7f8a6c06b760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8a84102f50 0x7f8a84113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 --2- 172.21.15.77:0/2886281920 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8a841038f0 0x7f8a84114020 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 >> 172.21.15.77:0/2886281920 conn(0x7f8a840f8ae0 msgr2=0x7f8a840fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 shutdown_connections 2024-02-19T18:57:11.167 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:11.149+0000 7f8a8a259700 1 -- 172.21.15.77:0/2886281920 wait complete. 2024-02-19T18:57:11.474 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:11 smithi077 ceph-mon[29247]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:11.474 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:11 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/2886281920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T18:57:11.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:11 smithi104 ceph-mon[42320]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:11.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:11 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/2886281920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T18:57:11.783 INFO:tasks.cephadm:Setting up client nodes... 2024-02-19T18:57:11.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-19T18:57:12.035 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:13.405 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:13 smithi077 ceph-mon[29247]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:13.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:13 smithi104 ceph-mon[42320]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:13.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.622+0000 7f6ee5d89700 1 -- 172.21.15.77:0/2767995862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fe0e0 msgr2=0x7f6ee00fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:13.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.622+0000 7f6ee5d89700 1 --2- 172.21.15.77:0/2767995862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fe0e0 0x7f6ee00fe550 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4004940 tx=0x7f6ed4009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:13.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.622+0000 7f6ee5d89700 1 -- 172.21.15.77:0/2767995862 shutdown_connections 2024-02-19T18:57:13.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.622+0000 7f6ee5d89700 1 --2- 172.21.15.77:0/2767995862 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fe0e0 0x7f6ee00fe550 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:13.642 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.622+0000 7f6ee5d89700 1 --2- 172.21.15.77:0/2767995862 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ee00fcef0 0x7f6ee00fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.622+0000 7f6ee5d89700 1 -- 172.21.15.77:0/2767995862 >> 172.21.15.77:0/2767995862 conn(0x7f6ee00f8620 msgr2=0x7f6ee00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.622+0000 7f6ee5d89700 1 -- 172.21.15.77:0/2767995862 shutdown_connections 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.622+0000 7f6ee5d89700 1 -- 172.21.15.77:0/2767995862 wait complete. 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6ee5d89700 1 Processor -- start 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6ee5d89700 1 -- start start 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6ee5d89700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fcef0 0x7f6ee0071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6ee5d89700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ee00fe0e0 0x7f6ee00702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6ee5d89700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0070970 con 0x7f6ee00fcef0 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6ee5d89700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee0070ae0 con 0x7f6ee00fe0e0 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6edf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fcef0 0x7f6ee0071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6edf7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fcef0 0x7f6ee0071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38860/0 (socket says 172.21.15.77:38860) 2024-02-19T18:57:13.643 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6edf7fe700 1 -- 172.21.15.77:0/4225720147 learned_addr learned my addr 172.21.15.77:0/4225720147 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6edeffd700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ee00fe0e0 0x7f6ee00702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6edf7fe700 1 -- 172.21.15.77:0/4225720147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ee00fe0e0 msgr2=0x7f6ee00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6edf7fe700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ee00fe0e0 0x7f6ee00702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6edf7fe700 1 -- 172.21.15.77:0/4225720147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed4005040 con 0x7f6ee00fcef0 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.623+0000 7f6edf7fe700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fcef0 0x7f6ee0071e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000bae0 tx=0x7f6ed0009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.624+0000 7f6ec7fff700 1 -- 172.21.15.77:0/4225720147 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ed000c8c0 con 0x7f6ee00fcef0 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.624+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee0070dc0 con 0x7f6ee00fcef0 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.624+0000 7f6ec7fff700 1 -- 172.21.15.77:0/4225720147 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6ed0014410 con 0x7f6ee00fcef0 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.624+0000 7f6ec7fff700 1 -- 172.21.15.77:0/4225720147 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6ed0007610 con 0x7f6ee00fcef0 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.624+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee0071140 con 0x7f6ee00fcef0 2024-02-19T18:57:13.644 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.625+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee004bed0 con 0x7f6ee00fcef0 2024-02-19T18:57:13.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.625+0000 7f6ec7fff700 1 -- 172.21.15.77:0/4225720147 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6ed0014580 con 0x7f6ee00fcef0 2024-02-19T18:57:13.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.625+0000 7f6ec7fff700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ec80692a0 0x7f6ec806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:13.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.625+0000 7f6ec7fff700 1 -- 172.21.15.77:0/4225720147 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6ed0086f50 con 0x7f6ee00fcef0 2024-02-19T18:57:13.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.626+0000 7f6edeffd700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ec80692a0 0x7f6ec806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:13.645 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.626+0000 7f6edeffd700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ec80692a0 0x7f6ec806b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4004f90 tx=0x7f6ed4009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:13.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.630+0000 7f6ec7fff700 1 -- 172.21.15.77:0/4225720147 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6ed004cbd0 con 0x7f6ee00fcef0 2024-02-19T18:57:13.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.895+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f6ee0105bc0 con 0x7f6ee00fcef0 2024-02-19T18:57:13.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.901+0000 7f6ec7fff700 1 -- 172.21.15.77:0/4225720147 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f6ed0056fd0 con 0x7f6ee00fcef0 2024-02-19T18:57:13.919 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-02-19T18:57:13.919 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQCJpNNltGJ8NRAA+8GpMQIagsoUdoy8lRhgqw== 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ec80692a0 msgr2=0x7f6ec806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ec80692a0 0x7f6ec806b750 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4004f90 tx=0x7f6ed4009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fcef0 msgr2=0x7f6ee0071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fcef0 0x7f6ee0071e60 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000bae0 tx=0x7f6ed0009810 comp rx=0 tx=0).stop 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 shutdown_connections 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ec80692a0 0x7f6ec806b750 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6ee00fcef0 0x7f6ee0071e60 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 --2- 172.21.15.77:0/4225720147 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6ee00fe0e0 0x7f6ee00702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 >> 172.21.15.77:0/4225720147 conn(0x7f6ee00f8620 msgr2=0x7f6ee01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 shutdown_connections 2024-02-19T18:57:13.921 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:13.903+0000 7f6ee5d89700 1 -- 172.21.15.77:0/4225720147 wait complete. 2024-02-19T18:57:14.480 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:14 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4225720147' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:57:14.480 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:14 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4225720147' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T18:57:14.504 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:57:14.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-19T18:57:14.504 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-19T18:57:14.546 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-19T18:57:14.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:14 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/4225720147' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:57:14.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:14 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/4225720147' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T18:57:14.809 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi104/config 2024-02-19T18:57:15.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:15 smithi077 ceph-mon[29247]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:15.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:15 smithi104 ceph-mon[42320]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:16.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.345+0000 7fc6f598a700 1 -- 172.21.15.104:0/1563722011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fe0e0 msgr2=0x7fc6f00fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:16.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.345+0000 7fc6f598a700 1 --2- 172.21.15.104:0/1563722011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fe0e0 0x7fc6f00fe550 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4004970 tx=0x7fc6e4009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:16.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.345+0000 7fc6f598a700 1 -- 172.21.15.104:0/1563722011 shutdown_connections 2024-02-19T18:57:16.364 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.345+0000 7fc6f598a700 1 --2- 172.21.15.104:0/1563722011 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fe0e0 0x7fc6f00fe550 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.345+0000 7fc6f598a700 1 --2- 172.21.15.104:0/1563722011 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6f00fcef0 0x7fc6f00fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.345+0000 7fc6f598a700 1 -- 172.21.15.104:0/1563722011 >> 172.21.15.104:0/1563722011 conn(0x7fc6f00f8620 msgr2=0x7fc6f00faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.346+0000 7fc6f598a700 1 -- 172.21.15.104:0/1563722011 shutdown_connections 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.346+0000 7fc6f598a700 1 -- 172.21.15.104:0/1563722011 wait complete. 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.346+0000 7fc6f598a700 1 Processor -- start 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.346+0000 7fc6f598a700 1 -- start start 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.346+0000 7fc6f598a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fcef0 0x7fc6f010d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.346+0000 7fc6f598a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6f00fe0e0 0x7fc6f010de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.346+0000 7fc6f598a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f010e450 con 0x7fc6f00fe0e0 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.346+0000 7fc6f598a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f010ecf0 con 0x7fc6f00fcef0 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6eeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fcef0 0x7fc6f010d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6eeffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fcef0 0x7fc6f010d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.104:49324/0 (socket says 172.21.15.104:49324) 2024-02-19T18:57:16.365 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6eeffd700 1 -- 172.21.15.104:0/3459683048 learned_addr learned my addr 172.21.15.104:0/3459683048 (peer_addr_for_me v2:172.21.15.104:0/0) 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6ee7fc700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6f00fe0e0 0x7fc6f010de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6eeffd700 1 -- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6f00fe0e0 msgr2=0x7fc6f010de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6eeffd700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6f00fe0e0 0x7fc6f010de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6eeffd700 1 -- 172.21.15.104:0/3459683048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e4005040 con 0x7fc6f00fcef0 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6eeffd700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fcef0 0x7fc6f010d910 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000c8e0 tx=0x7fc6e00044c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6d77fe700 1 -- 172.21.15.104:0/3459683048 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6e000a940 con 0x7fc6f00fcef0 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6d77fe700 1 -- 172.21.15.104:0/3459683048 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc6e000ce80 con 0x7fc6f00fcef0 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f010efd0 con 0x7fc6f00fcef0 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6d77fe700 1 -- 172.21.15.104:0/3459683048 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc6e00113a0 con 0x7fc6f00fcef0 2024-02-19T18:57:16.366 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.347+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f010f510 con 0x7fc6f00fcef0 2024-02-19T18:57:16.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.348+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f004bed0 con 0x7fc6f00fcef0 2024-02-19T18:57:16.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.349+0000 7fc6d77fe700 1 -- 172.21.15.104:0/3459683048 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc6e0012400 con 0x7fc6f00fcef0 2024-02-19T18:57:16.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.349+0000 7fc6d77fe700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc6d80692a0 0x7fc6d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:16.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.349+0000 7fc6d77fe700 1 -- 172.21.15.104:0/3459683048 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc6e0086e40 con 0x7fc6f00fcef0 2024-02-19T18:57:16.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.349+0000 7fc6ee7fc700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc6d80692a0 0x7fc6d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:16.367 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.350+0000 7fc6ee7fc700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc6d80692a0 0x7fc6d806b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4000c00 tx=0x7fc6e4009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:16.370 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.353+0000 7fc6d77fe700 1 -- 172.21.15.104:0/3459683048 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc6e0056fe0 con 0x7fc6f00fcef0 2024-02-19T18:57:16.625 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.608+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc6f0105a30 con 0x7fc6f00fcef0 2024-02-19T18:57:16.630 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.613+0000 7fc6d77fe700 1 -- 172.21.15.104:0/3459683048 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc6e0056b70 con 0x7fc6f00fcef0 2024-02-19T18:57:16.630 INFO:teuthology.orchestra.run.smithi104.stdout:[client.1] 2024-02-19T18:57:16.630 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQCMpNNlW5xYJBAAnsN+382UfoIODJsLa/9Nlw== 2024-02-19T18:57:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc6d80692a0 msgr2=0x7fc6d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc6d80692a0 0x7fc6d806b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc6e4000c00 tx=0x7fc6e4009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fcef0 msgr2=0x7fc6f010d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fcef0 0x7fc6f010d910 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000c8e0 tx=0x7fc6e00044c0 comp rx=0 tx=0).stop 2024-02-19T18:57:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 shutdown_connections 2024-02-19T18:57:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc6d80692a0 0x7fc6d806b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:16.632 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc6f00fcef0 0x7fc6f010d910 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 --2- 172.21.15.104:0/3459683048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc6f00fe0e0 0x7fc6f010de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 >> 172.21.15.104:0/3459683048 conn(0x7fc6f00f8620 msgr2=0x7fc6f01013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 shutdown_connections 2024-02-19T18:57:16.633 INFO:teuthology.orchestra.run.smithi104.stderr:2024-02-19T18:57:16.615+0000 7fc6f598a700 1 -- 172.21.15.104:0/3459683048 wait complete. 2024-02-19T18:57:17.326 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:57:17.326 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-19T18:57:17.327 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-19T18:57:17.410 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-19T18:57:17.410 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-19T18:57:17.410 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph mgr dump --format=json 2024-02-19T18:57:17.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:17 smithi077 ceph-mon[29247]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:17.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:17 smithi077 ceph-mon[29247]: from='client.? 172.21.15.104:0/3459683048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:57:17.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:17 smithi077 ceph-mon[29247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:57:17.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:17 smithi077 ceph-mon[29247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T18:57:17.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:17 smithi104 ceph-mon[42320]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:17.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:17 smithi104 ceph-mon[42320]: from='client.? 172.21.15.104:0/3459683048' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:57:17.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:17 smithi104 ceph-mon[42320]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-19T18:57:17.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:17 smithi104 ceph-mon[42320]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-19T18:57:17.673 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:19.035 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:18 smithi077 ceph-mon[29247]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.243+0000 7fae81074700 1 -- 172.21.15.77:0/2753319020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 msgr2=0x7fae7c0fe330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.243+0000 7fae81074700 1 --2- 172.21.15.77:0/2753319020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 0x7fae7c0fe330 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fae6c004940 tx=0x7fae6c009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.243+0000 7fae81074700 1 -- 172.21.15.77:0/2753319020 shutdown_connections 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.243+0000 7fae81074700 1 --2- 172.21.15.77:0/2753319020 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe900 0x7fae7c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.243+0000 7fae81074700 1 --2- 172.21.15.77:0/2753319020 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 0x7fae7c0fe330 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.243+0000 7fae81074700 1 -- 172.21.15.77:0/2753319020 >> 172.21.15.77:0/2753319020 conn(0x7fae7c0f7a10 msgr2=0x7fae7c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.243+0000 7fae81074700 1 -- 172.21.15.77:0/2753319020 shutdown_connections 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.243+0000 7fae81074700 1 -- 172.21.15.77:0/2753319020 wait complete. 2024-02-19T18:57:19.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae81074700 1 Processor -- start 2024-02-19T18:57:19.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae81074700 1 -- start start 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae81074700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 0x7fae7c113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae81074700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe900 0x7fae7c114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae81074700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c1146a0 con 0x7fae7c0fdf60 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae81074700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7c10d9d0 con 0x7fae7c0fe900 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe900 0x7fae7c114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7ad9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 0x7fae7c113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7a59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe900 0x7fae7c114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.77:43698/0 (socket says 172.21.15.77:43698) 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7a59c700 1 -- 172.21.15.77:0/3606014179 learned_addr learned my addr 172.21.15.77:0/3606014179 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7ad9d700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 0x7fae7c113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43572/0 (socket says 172.21.15.77:43572) 2024-02-19T18:57:19.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7ad9d700 1 -- 172.21.15.77:0/3606014179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe900 msgr2=0x7fae7c114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7ad9d700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe900 0x7fae7c114050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7ad9d700 1 -- 172.21.15.77:0/3606014179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c005040 con 0x7fae7c0fdf60 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7a59c700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe900 0x7fae7c114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae7ad9d700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 0x7fae7c113b10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000c00 tx=0x7fae6c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.244+0000 7fae637fe700 1 -- 172.21.15.77:0/3606014179 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae6c021070 con 0x7fae7c0fdf60 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.245+0000 7fae637fe700 1 -- 172.21.15.77:0/3606014179 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fae6c013d00 con 0x7fae7c0fdf60 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.245+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7c10dc50 con 0x7fae7c0fdf60 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.245+0000 7fae637fe700 1 -- 172.21.15.77:0/3606014179 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae6c00e070 con 0x7fae7c0fdf60 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.245+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7c10e190 con 0x7fae7c0fdf60 2024-02-19T18:57:19.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.245+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7c104550 con 0x7fae7c0fdf60 2024-02-19T18:57:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.246+0000 7fae637fe700 1 -- 172.21.15.77:0/3606014179 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fae6c01cbd0 con 0x7fae7c0fdf60 2024-02-19T18:57:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.246+0000 7fae637fe700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fae640692a0 0x7fae6406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.246+0000 7fae637fe700 1 -- 172.21.15.77:0/3606014179 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fae6c088250 con 0x7fae7c0fdf60 2024-02-19T18:57:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.246+0000 7fae7a59c700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fae640692a0 0x7fae6406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:19.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.247+0000 7fae7a59c700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fae640692a0 0x7fae6406b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fae7c10f340 tx=0x7fae7000b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:19.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.251+0000 7fae637fe700 1 -- 172.21.15.77:0/3606014179 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fae6c058410 con 0x7fae7c0fdf60 2024-02-19T18:57:19.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:18 smithi104 ceph-mon[42320]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:19.503 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.486+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fae7c10a080 con 0x7fae7c0fdf60 2024-02-19T18:57:19.506 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.489+0000 7fae637fe700 1 -- 172.21.15.77:0/3606014179 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167170 (secure 0 0 0) 0x7fae6c057fa0 con 0x7fae7c0fdf60 2024-02-19T18:57:19.509 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:19.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fae640692a0 msgr2=0x7fae6406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:19.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fae640692a0 0x7fae6406b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fae7c10f340 tx=0x7fae7000b040 comp rx=0 tx=0).stop 2024-02-19T18:57:19.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 msgr2=0x7fae7c113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 0x7fae7c113b10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fae6c000c00 tx=0x7fae6c009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 shutdown_connections 2024-02-19T18:57:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fae640692a0 0x7fae6406b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fae7c0fdf60 0x7fae7c113b10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 --2- 172.21.15.77:0/3606014179 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fae7c0fe900 0x7fae7c114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 >> 172.21.15.77:0/3606014179 conn(0x7fae7c0f7a10 msgr2=0x7fae7c1016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 shutdown_connections 2024-02-19T18:57:19.511 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:19.491+0000 7fae81074700 1 -- 172.21.15.77:0/3606014179 wait complete. 2024-02-19T18:57:20.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:20.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:19 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3606014179' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T18:57:20.169 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:20.169 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:19 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3606014179' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-19T18:57:20.194 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":19,"active_gid":14270,"active_name":"smithi077.yiiqpd","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":2462366592},{"type":"v1","addr":"172.21.15.77:6801","nonce":2462366592}]},"active_addr":"172.21.15.77:6801/2462366592","active_change":"2024-02-19T18:53:34.087153+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14292,"name":"smithi104.sdpzmi","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.77:8443/","prometheus":"http://172.21.15.77:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2181676682}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2586414670}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":989226431}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.77:0","nonce":2921806564}]}]} 2024-02-19T18:57:20.198 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-19T18:57:20.198 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-19T18:57:20.198 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T18:57:20.450 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:21.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:20 smithi077 ceph-mon[29247]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:21.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:20 smithi104 ceph-mon[42320]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.099+0000 7f06a0aef700 1 -- 172.21.15.77:0/3312699855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 msgr2=0x7f069c103310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.099+0000 7f06a0aef700 1 --2- 172.21.15.77:0/3312699855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 0x7f069c103310 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f068c004940 tx=0x7f068c009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.099+0000 7f06a0aef700 1 -- 172.21.15.77:0/3312699855 shutdown_connections 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.099+0000 7f06a0aef700 1 --2- 172.21.15.77:0/3312699855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f069c1038e0 0x7f069c0fcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.099+0000 7f06a0aef700 1 --2- 172.21.15.77:0/3312699855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 0x7f069c103310 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.099+0000 7f06a0aef700 1 -- 172.21.15.77:0/3312699855 >> 172.21.15.77:0/3312699855 conn(0x7f069c0f8af0 msgr2=0x7f069c0faf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.100+0000 7f06a0aef700 1 -- 172.21.15.77:0/3312699855 shutdown_connections 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.100+0000 7f06a0aef700 1 -- 172.21.15.77:0/3312699855 wait complete. 2024-02-19T18:57:22.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.100+0000 7f06a0aef700 1 Processor -- start 2024-02-19T18:57:22.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.100+0000 7f06a0aef700 1 -- start start 2024-02-19T18:57:22.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.100+0000 7f06a0aef700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 0x7f069c113ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:22.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.100+0000 7f06a0aef700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f069c1038e0 0x7f069c114010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:22.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.100+0000 7f06a0aef700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c1146f0 con 0x7f069c102f40 2024-02-19T18:57:22.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.100+0000 7f06a0aef700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c10d990 con 0x7f069c1038e0 2024-02-19T18:57:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f069a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 0x7f069c113ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f069a59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 0x7f069c113ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43594/0 (socket says 172.21.15.77:43594) 2024-02-19T18:57:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f069a59c700 1 -- 172.21.15.77:0/566085154 learned_addr learned my addr 172.21.15.77:0/566085154 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f0699d9b700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f069c1038e0 0x7f069c114010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f069a59c700 1 -- 172.21.15.77:0/566085154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f069c1038e0 msgr2=0x7f069c114010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f069a59c700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f069c1038e0 0x7f069c114010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f069a59c700 1 -- 172.21.15.77:0/566085154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f068c005040 con 0x7f069c102f40 2024-02-19T18:57:22.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f0699d9b700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f069c1038e0 0x7f069c114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:57:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f069a59c700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 0x7f069c113ad0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f068c00b3b0 tx=0x7f068c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f068affd700 1 -- 172.21.15.77:0/566085154 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f068c00e040 con 0x7f069c102f40 2024-02-19T18:57:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f068affd700 1 -- 172.21.15.77:0/566085154 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f068c00bbf0 con 0x7f069c102f40 2024-02-19T18:57:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c10dc10 con 0x7f069c102f40 2024-02-19T18:57:22.122 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f068affd700 1 -- 172.21.15.77:0/566085154 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f068c01c630 con 0x7f069c102f40 2024-02-19T18:57:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.101+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c10e150 con 0x7f069c102f40 2024-02-19T18:57:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.102+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c0fc6f0 con 0x7f069c102f40 2024-02-19T18:57:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.103+0000 7f068affd700 1 -- 172.21.15.77:0/566085154 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f068c00bd60 con 0x7f069c102f40 2024-02-19T18:57:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.103+0000 7f068affd700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f06840692a0 0x7f068406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.103+0000 7f0699d9b700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f06840692a0 0x7f068406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.103+0000 7f068affd700 1 -- 172.21.15.77:0/566085154 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f068c088200 con 0x7f069c102f40 2024-02-19T18:57:22.123 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.104+0000 7f0699d9b700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f06840692a0 0x7f068406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f069c10f300 tx=0x7f0690008040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:22.125 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.108+0000 7f068affd700 1 -- 172.21.15.77:0/566085154 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f068c058300 con 0x7f069c102f40 2024-02-19T18:57:22.308 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.290+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f069c10f0b0 con 0x7f069c102f40 2024-02-19T18:57:22.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.291+0000 7f068affd700 1 -- 172.21.15.77:0/566085154 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11072 (secure 0 0 0) 0x7f068c057e90 con 0x7f069c102f40 2024-02-19T18:57:22.309 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:22.309 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","created":"2024-02-19T18:49:39.682097+0000","modified":"2024-02-19T18:57:03.313927+0000","last_up_change":"2024-02-19T18:57:02.307357+0000","last_in_change":"2024-02-19T18:56:42.906075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:55:48.412324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fbc9e22d-2cb7-4421-b6d8-0dac7325974f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6803","nonce":806127290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6805","nonce":806127290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6809","nonce":806127290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6807","nonce":806127290}]},"public_addr":"172.21.15.77:6803/806127290","cluster_addr":"172.21.15.77:6805/806127290","heartbeat_back_addr":"172.21.15.77:6809/806127290","heartbeat_front_addr":"172.21.15.77:6807/806127290","state":["exists","up"]},{"osd":1,"uuid":"c544f75d-7ae5-4261-b55e-f2be62afb9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6811","nonce":3114250568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6813","nonce":3114250568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6817","nonce":3114250568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6815","nonce":3114250568}]},"public_addr":"172.21.15.77:6811/3114250568","cluster_addr":"172.21.15.77:6813/3114250568","heartbeat_back_addr":"172.21.15.77:6817/3114250568","heartbeat_front_addr":"172.21.15.77:6815/3114250568","state":["exists","up"]},{"osd":2,"uuid":"8463e27a-7421-42a1-8b87-7185b9f61766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6819","nonce":100949402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6821","nonce":100949402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6825","nonce":100949402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6823","nonce":100949402}]},"public_addr":"172.21.15.77:6819/100949402","cluster_addr":"172.21.15.77:6821/100949402","heartbeat_back_addr":"172.21.15.77:6825/100949402","heartbeat_front_addr":"172.21.15.77:6823/100949402","state":["exists","up"]},{"osd":3,"uuid":"d15c9aad-95fd-4446-b3da-4f447fc3f7a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6801","nonce":3308675957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6803","nonce":3308675957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6807","nonce":3308675957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6805","nonce":3308675957}]},"public_addr":"172.21.15.104:6801/3308675957","cluster_addr":"172.21.15.104:6803/3308675957","heartbeat_back_addr":"172.21.15.104:6807/3308675957","heartbeat_front_addr":"172.21.15.104:6805/3308675957","state":["exists","up"]},{"osd":4,"uuid":"a37971a1-f37c-4136-ae69-8a9999870db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6809","nonce":3596569551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6811","nonce":3596569551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6815","nonce":3596569551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6813","nonce":3596569551}]},"public_addr":"172.21.15.104:6809/3596569551","cluster_addr":"172.21.15.104:6811/3596569551","heartbeat_back_addr":"172.21.15.104:6815/3596569551","heartbeat_front_addr":"172.21.15.104:6813/3596569551","state":["exists","up"]},{"osd":5,"uuid":"ffc028a3-833b-4811-a38b-a47f1c6e15da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6817","nonce":2114623660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6819","nonce":2114623660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6823","nonce":2114623660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6821","nonce":2114623660}]},"public_addr":"172.21.15.104:6817/2114623660","cluster_addr":"172.21.15.104:6819/2114623660","heartbeat_back_addr":"172.21.15.104:6823/2114623660","heartbeat_front_addr":"172.21.15.104:6821/2114623660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:54:46.893755+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:55:15.359473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:55:45.118036+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:08.973267+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:34.107665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:00.315125+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2151174911":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/352143230":"2024-02-20T18:53:34.087006+0000","172.21.15.77:6800/3287530536":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/4032060278":"2024-02-20T18:53:34.087006+0000","172.21.15.77:6801/1028404574":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/2030253411":"2024-02-20T18:50:15.313914+0000","172.21.15.77:0/3240479911":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/2766229332":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/1552197174":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6800/1028404574":"2024-02-20T18:50:46.166039+0000","172.21.15.77:6801/2289192383":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6801/3287530536":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/1650327672":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/1405110191":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6800/2289192383":"2024-02-20T18:50:15.313914+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T18:57:22.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f06840692a0 msgr2=0x7f068406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:22.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f06840692a0 0x7f068406b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f069c10f300 tx=0x7f0690008040 comp rx=0 tx=0).stop 2024-02-19T18:57:22.311 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 msgr2=0x7f069c113ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:22.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 0x7f069c113ad0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f068c00b3b0 tx=0x7f068c009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:22.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 shutdown_connections 2024-02-19T18:57:22.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f06840692a0 0x7f068406b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:22.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f069c102f40 0x7f069c113ad0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:22.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 --2- 172.21.15.77:0/566085154 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f069c1038e0 0x7f069c114010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:22.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 >> 172.21.15.77:0/566085154 conn(0x7f069c0f8af0 msgr2=0x7f069c0fa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:22.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 shutdown_connections 2024-02-19T18:57:22.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:22.293+0000 7f06a0aef700 1 -- 172.21.15.77:0/566085154 wait complete. 2024-02-19T18:57:22.886 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-19T18:57:22.886 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd dump --format=json 2024-02-19T18:57:23.141 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:23.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:23 smithi077 ceph-mon[29247]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:23.528 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:23 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/566085154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:57:23.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:23 smithi104 ceph-mon[42320]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:23.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:23 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/566085154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:57:24.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.638+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3782304387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fe8f0 msgr2=0x7f3d04107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.638+0000 7f3d0bfb2700 1 --2- 172.21.15.77:0/3782304387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fe8f0 0x7f3d04107050 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3cf4004970 tx=0x7f3cf4009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.638+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3782304387 shutdown_connections 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.638+0000 7f3d0bfb2700 1 --2- 172.21.15.77:0/3782304387 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fe8f0 0x7f3d04107050 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.638+0000 7f3d0bfb2700 1 --2- 172.21.15.77:0/3782304387 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d040fdfe0 0x7f3d040fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.638+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3782304387 >> 172.21.15.77:0/3782304387 conn(0x7f3d04071d90 msgr2=0x7f3d04072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3782304387 shutdown_connections 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3782304387 wait complete. 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d0bfb2700 1 Processor -- start 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d0bfb2700 1 -- start start 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d0bfb2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fdfe0 0x7f3d04113a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:24.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d0bfb2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d040fe8f0 0x7f3d04113fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d0bfb2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d04114690 con 0x7f3d040fdfe0 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d0bfb2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0410d930 con 0x7f3d040fe8f0 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d09d4e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fdfe0 0x7f3d04113a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d09d4e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fdfe0 0x7f3d04113a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:43600/0 (socket says 172.21.15.77:43600) 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.639+0000 7f3d09d4e700 1 -- 172.21.15.77:0/3843227741 learned_addr learned my addr 172.21.15.77:0/3843227741 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3d0954d700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d040fe8f0 0x7f3d04113fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3d09d4e700 1 -- 172.21.15.77:0/3843227741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d040fe8f0 msgr2=0x7f3d04113fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3d09d4e700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d040fe8f0 0x7f3d04113fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3d09d4e700 1 -- 172.21.15.77:0/3843227741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf4005040 con 0x7f3d040fdfe0 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3d0954d700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d040fe8f0 0x7f3d04113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3d09d4e700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fdfe0 0x7f3d04113a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000b790 tx=0x7f3d00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3cfa7fc700 1 -- 172.21.15.77:0/3843227741 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d0000f770 con 0x7f3d040fdfe0 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0410dc10 con 0x7f3d040fdfe0 2024-02-19T18:57:24.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3cfa7fc700 1 -- 172.21.15.77:0/3843227741 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3d0000fda0 con 0x7f3d040fdfe0 2024-02-19T18:57:24.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3cfa7fc700 1 -- 172.21.15.77:0/3843227741 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3d0000d310 con 0x7f3d040fdfe0 2024-02-19T18:57:24.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.640+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d0410e120 con 0x7f3d040fdfe0 2024-02-19T18:57:24.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.641+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d041046c0 con 0x7f3d040fdfe0 2024-02-19T18:57:24.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.642+0000 7f3cfa7fc700 1 -- 172.21.15.77:0/3843227741 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3d00010420 con 0x7f3d040fdfe0 2024-02-19T18:57:24.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.642+0000 7f3cfa7fc700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3cf00692a0 0x7f3cf006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:24.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.642+0000 7f3cfa7fc700 1 -- 172.21.15.77:0/3843227741 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3d00086e70 con 0x7f3d040fdfe0 2024-02-19T18:57:24.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.642+0000 7f3d0954d700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3cf00692a0 0x7f3cf006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:24.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.642+0000 7f3d0954d700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3cf00692a0 0x7f3cf006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3d0410ec60 tx=0x7f3cf4009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:24.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.646+0000 7f3cfa7fc700 1 -- 172.21.15.77:0/3843227741 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3d00056f70 con 0x7f3d040fdfe0 2024-02-19T18:57:24.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.828+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d040738b0 con 0x7f3d040fdfe0 2024-02-19T18:57:24.846 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.829+0000 7f3cfa7fc700 1 -- 172.21.15.77:0/3843227741 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11072 (secure 0 0 0) 0x7f3d00056b00 con 0x7f3d040fdfe0 2024-02-19T18:57:24.847 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:24.847 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":41,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","created":"2024-02-19T18:49:39.682097+0000","modified":"2024-02-19T18:57:03.313927+0000","last_up_change":"2024-02-19T18:57:02.307357+0000","last_in_change":"2024-02-19T18:56:42.906075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:55:48.412324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fbc9e22d-2cb7-4421-b6d8-0dac7325974f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6803","nonce":806127290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6805","nonce":806127290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6809","nonce":806127290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6807","nonce":806127290}]},"public_addr":"172.21.15.77:6803/806127290","cluster_addr":"172.21.15.77:6805/806127290","heartbeat_back_addr":"172.21.15.77:6809/806127290","heartbeat_front_addr":"172.21.15.77:6807/806127290","state":["exists","up"]},{"osd":1,"uuid":"c544f75d-7ae5-4261-b55e-f2be62afb9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6811","nonce":3114250568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6813","nonce":3114250568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6817","nonce":3114250568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6815","nonce":3114250568}]},"public_addr":"172.21.15.77:6811/3114250568","cluster_addr":"172.21.15.77:6813/3114250568","heartbeat_back_addr":"172.21.15.77:6817/3114250568","heartbeat_front_addr":"172.21.15.77:6815/3114250568","state":["exists","up"]},{"osd":2,"uuid":"8463e27a-7421-42a1-8b87-7185b9f61766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6819","nonce":100949402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6821","nonce":100949402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6825","nonce":100949402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6823","nonce":100949402}]},"public_addr":"172.21.15.77:6819/100949402","cluster_addr":"172.21.15.77:6821/100949402","heartbeat_back_addr":"172.21.15.77:6825/100949402","heartbeat_front_addr":"172.21.15.77:6823/100949402","state":["exists","up"]},{"osd":3,"uuid":"d15c9aad-95fd-4446-b3da-4f447fc3f7a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6801","nonce":3308675957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6803","nonce":3308675957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6807","nonce":3308675957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6805","nonce":3308675957}]},"public_addr":"172.21.15.104:6801/3308675957","cluster_addr":"172.21.15.104:6803/3308675957","heartbeat_back_addr":"172.21.15.104:6807/3308675957","heartbeat_front_addr":"172.21.15.104:6805/3308675957","state":["exists","up"]},{"osd":4,"uuid":"a37971a1-f37c-4136-ae69-8a9999870db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6809","nonce":3596569551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6811","nonce":3596569551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6815","nonce":3596569551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6813","nonce":3596569551}]},"public_addr":"172.21.15.104:6809/3596569551","cluster_addr":"172.21.15.104:6811/3596569551","heartbeat_back_addr":"172.21.15.104:6815/3596569551","heartbeat_front_addr":"172.21.15.104:6813/3596569551","state":["exists","up"]},{"osd":5,"uuid":"ffc028a3-833b-4811-a38b-a47f1c6e15da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6817","nonce":2114623660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6819","nonce":2114623660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6823","nonce":2114623660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6821","nonce":2114623660}]},"public_addr":"172.21.15.104:6817/2114623660","cluster_addr":"172.21.15.104:6819/2114623660","heartbeat_back_addr":"172.21.15.104:6823/2114623660","heartbeat_front_addr":"172.21.15.104:6821/2114623660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:54:46.893755+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:55:15.359473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:55:45.118036+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:08.973267+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:34.107665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:00.315125+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2151174911":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/352143230":"2024-02-20T18:53:34.087006+0000","172.21.15.77:6800/3287530536":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/4032060278":"2024-02-20T18:53:34.087006+0000","172.21.15.77:6801/1028404574":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/2030253411":"2024-02-20T18:50:15.313914+0000","172.21.15.77:0/3240479911":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/2766229332":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/1552197174":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6800/1028404574":"2024-02-20T18:50:46.166039+0000","172.21.15.77:6801/2289192383":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6801/3287530536":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/1650327672":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/1405110191":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6800/2289192383":"2024-02-20T18:50:15.313914+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3cf00692a0 msgr2=0x7f3cf006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3cf00692a0 0x7f3cf006b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3d0410ec60 tx=0x7f3cf4009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fdfe0 msgr2=0x7f3d04113a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fdfe0 0x7f3d04113a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3d0000b790 tx=0x7f3d00009250 comp rx=0 tx=0).stop 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 shutdown_connections 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3cf00692a0 0x7f3cf006b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3d040fdfe0 0x7f3d04113a70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 --2- 172.21.15.77:0/3843227741 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3d040fe8f0 0x7f3d04113fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 >> 172.21.15.77:0/3843227741 conn(0x7f3d04071d90 msgr2=0x7f3d04101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 shutdown_connections 2024-02-19T18:57:24.849 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:24.831+0000 7f3d0bfb2700 1 -- 172.21.15.77:0/3843227741 wait complete. 2024-02-19T18:57:25.453 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:25 smithi077 ceph-mon[29247]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:25.453 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:25 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3843227741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:57:25.455 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-19T18:57:25.455 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-19T18:57:25.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-19T18:57:25.456 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-19T18:57:25.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-19T18:57:25.457 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-19T18:57:25.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:25 smithi104 ceph-mon[42320]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:25.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:25 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3843227741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:57:25.753 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:25.762 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:26.519 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:26.521 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:27.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:27 smithi077 ceph-mon[29247]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:27.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:27 smithi104 ceph-mon[42320]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:28.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:28 smithi077 ceph-mon[29247]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:29.182 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:29.263 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:28 smithi104 ceph-mon[42320]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.237+0000 7f1d8c481700 1 -- 172.21.15.77:0/3498003836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 msgr2=0x7f1d84103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.237+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3498003836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 0x7f1d84103320 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f1d80004940 tx=0x7f1d80009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.237+0000 7f1d8c481700 1 -- 172.21.15.77:0/3498003836 shutdown_connections 2024-02-19T18:57:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.237+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3498003836 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d841038f0 0x7f1d840fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.237+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3498003836 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 0x7f1d84103320 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.237+0000 7f1d8c481700 1 -- 172.21.15.77:0/3498003836 >> 172.21.15.77:0/3498003836 conn(0x7f1d840f8ae0 msgr2=0x7f1d840faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:30.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.237+0000 7f1d8c481700 1 -- 172.21.15.77:0/3498003836 shutdown_connections 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8c481700 1 -- 172.21.15.77:0/3498003836 wait complete. 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8c481700 1 Processor -- start 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8c481700 1 -- start start 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8c481700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 0x7f1d84070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8c481700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d841038f0 0x7f1d840707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8c481700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d84070e60 con 0x7f1d84102f50 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8c481700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d84070fd0 con 0x7f1d841038f0 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8a21d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 0x7f1d84070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8a21d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 0x7f1d84070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47816/0 (socket says 172.21.15.77:47816) 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d8a21d700 1 -- 172.21.15.77:0/3163196832 learned_addr learned my addr 172.21.15.77:0/3163196832 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.238+0000 7f1d89a1c700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d841038f0 0x7f1d840707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d8a21d700 1 -- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d841038f0 msgr2=0x7f1d840707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d8a21d700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d841038f0 0x7f1d840707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d8a21d700 1 -- 172.21.15.77:0/3163196832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d80005040 con 0x7f1d84102f50 2024-02-19T18:57:30.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d89a1c700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d841038f0 0x7f1d840707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:57:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d8a21d700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 0x7f1d84070290 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d80009fd0 tx=0x7f1d80009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d8000e040 con 0x7f1d84102f50 2024-02-19T18:57:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d84119d20 con 0x7f1d84102f50 2024-02-19T18:57:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1d8000bbf0 con 0x7f1d84102f50 2024-02-19T18:57:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1d8001c3f0 con 0x7f1d84102f50 2024-02-19T18:57:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.239+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8411a210 con 0x7f1d84102f50 2024-02-19T18:57:30.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.240+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1d840fadd0 con 0x7f1d84102f50 2024-02-19T18:57:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.240+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1d8000bd60 con 0x7f1d84102f50 2024-02-19T18:57:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d7affd700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1d700692b0 0x7f1d7006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1d80088bf0 con 0x7f1d84102f50 2024-02-19T18:57:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d89a1c700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1d700692b0 0x7f1d7006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d7affd700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] conn(0x7f1d7006ef80 0x7f1d70071390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 --> [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d70071a40 con 0x7f1d7006ef80 2024-02-19T18:57:30.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d80088fd0 con 0x7f1d84102f50 2024-02-19T18:57:30.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d89a1c700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1d700692b0 0x7f1d7006b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1d74006fd0 tx=0x7f1d74008040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:30.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d8aa1e700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] conn(0x7f1d7006ef80 0x7f1d70071390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:30.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.241+0000 7f1d8aa1e700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] conn(0x7f1d7006ef80 0x7f1d70071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:30.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.242+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 <== osd.3 v2:172.21.15.104:6800/3308675957 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f1d70071a40 con 0x7f1d7006ef80 2024-02-19T18:57:30.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.257+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 --> [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1d840fc6e0 con 0x7f1d7006ef80 2024-02-19T18:57:30.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.257+0000 7f1d7affd700 1 -- 172.21.15.77:0/3163196832 <== osd.3 v2:172.21.15.104:6800/3308675957 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1d840fc6e0 con 0x7f1d7006ef80 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] conn(0x7f1d7006ef80 msgr2=0x7f1d70071390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] conn(0x7f1d7006ef80 0x7f1d70071390 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1d700692b0 msgr2=0x7f1d7006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1d700692b0 0x7f1d7006b760 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1d74006fd0 tx=0x7f1d74008040 comp rx=0 tx=0).stop 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 msgr2=0x7f1d84070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 0x7f1d84070290 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1d80009fd0 tx=0x7f1d80009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 shutdown_connections 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:6800/3308675957,v1:172.21.15.104:6801/3308675957] conn(0x7f1d7006ef80 0x7f1d70071390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1d700692b0 0x7f1d7006b760 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1d84102f50 0x7f1d84070290 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:30.276 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 --2- 172.21.15.77:0/3163196832 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1d841038f0 0x7f1d840707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:30.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 >> 172.21.15.77:0/3163196832 conn(0x7f1d840f8ae0 msgr2=0x7f1d840fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:30.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 shutdown_connections 2024-02-19T18:57:30.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:30.258+0000 7f1d8c481700 1 -- 172.21.15.77:0/3163196832 wait complete. 2024-02-19T18:57:30.920 INFO:teuthology.orchestra.run.smithi077.stdout:115964117010 2024-02-19T18:57:30.920 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-19T18:57:31.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:31 smithi077 ceph-mon[29247]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:31.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:31 smithi104 ceph-mon[42320]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:31.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.854+0000 7f313eae4700 1 -- 172.21.15.77:0/3085698750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fe8f0 msgr2=0x7f3138107050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:31.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.854+0000 7f313eae4700 1 --2- 172.21.15.77:0/3085698750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fe8f0 0x7f3138107050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3124004970 tx=0x7f3124009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:31.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.854+0000 7f313eae4700 1 -- 172.21.15.77:0/3085698750 shutdown_connections 2024-02-19T18:57:31.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.854+0000 7f313eae4700 1 --2- 172.21.15.77:0/3085698750 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fe8f0 0x7f3138107050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.854+0000 7f313eae4700 1 --2- 172.21.15.77:0/3085698750 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31380fdfe0 0x7f31380fe3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.854+0000 7f313eae4700 1 -- 172.21.15.77:0/3085698750 >> 172.21.15.77:0/3085698750 conn(0x7f3138071d90 msgr2=0x7f3138072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.854+0000 7f313eae4700 1 -- 172.21.15.77:0/3085698750 shutdown_connections 2024-02-19T18:57:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.854+0000 7f313eae4700 1 -- 172.21.15.77:0/3085698750 wait complete. 2024-02-19T18:57:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313eae4700 1 Processor -- start 2024-02-19T18:57:31.873 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313eae4700 1 -- start start 2024-02-19T18:57:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313eae4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fdfe0 0x7f31380fbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313eae4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31380fe8f0 0x7f31380fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313eae4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380fc9e0 con 0x7f31380fdfe0 2024-02-19T18:57:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313eae4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31380fcb50 con 0x7f31380fe8f0 2024-02-19T18:57:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313c880700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fdfe0 0x7f31380fbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313c880700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fdfe0 0x7f31380fbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47838/0 (socket says 172.21.15.77:47838) 2024-02-19T18:57:31.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313c880700 1 -- 172.21.15.77:0/3986763576 learned_addr learned my addr 172.21.15.77:0/3986763576 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f3137fff700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31380fe8f0 0x7f31380fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313c880700 1 -- 172.21.15.77:0/3986763576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31380fe8f0 msgr2=0x7f31380fc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313c880700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31380fe8f0 0x7f31380fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313c880700 1 -- 172.21.15.77:0/3986763576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124005040 con 0x7f31380fdfe0 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.855+0000 7f313c880700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fdfe0 0x7f31380fbea0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f312c00c830 tx=0x7f312c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.856+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f312c00a760 con 0x7f31380fdfe0 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.856+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f312c00ad90 con 0x7f31380fdfe0 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.856+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3138119c30 con 0x7f31380fdfe0 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.856+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f312c0113a0 con 0x7f31380fdfe0 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.856+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313811a110 con 0x7f31380fdfe0 2024-02-19T18:57:31.875 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.856+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f313805e650 con 0x7f31380fdfe0 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.857+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f312c012400 con 0x7f31380fdfe0 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.857+0000 7f31357fa700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31280692a0 0x7f312806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.857+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f312c086eb0 con 0x7f31380fdfe0 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.857+0000 7f3137fff700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31280692a0 0x7f312806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.858+0000 7f31357fa700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] conn(0x7f312806ef70 0x7f3128071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.858+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 --> [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3128071a30 con 0x7f312806ef70 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.858+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f312c08c4b0 con 0x7f31380fdfe0 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.858+0000 7f313d081700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] conn(0x7f312806ef70 0x7f3128071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.858+0000 7f3137fff700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31280692a0 0x7f312806b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3124009fd0 tx=0x7f3124009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.858+0000 7f313d081700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] conn(0x7f312806ef70 0x7f3128071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:31.877 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.859+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 <== osd.1 v2:172.21.15.77:6810/3114250568 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3128071a30 con 0x7f312806ef70 2024-02-19T18:57:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.873+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 --> [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f313805ead0 con 0x7f312806ef70 2024-02-19T18:57:31.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f31357fa700 1 -- 172.21.15.77:0/3986763576 <== osd.1 v2:172.21.15.77:6810/3114250568 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f313805ead0 con 0x7f312806ef70 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] conn(0x7f312806ef70 msgr2=0x7f3128071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] conn(0x7f312806ef70 0x7f3128071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31280692a0 msgr2=0x7f312806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31280692a0 0x7f312806b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3124009fd0 tx=0x7f3124009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fdfe0 msgr2=0x7f31380fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fdfe0 0x7f31380fbea0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f312c00c830 tx=0x7f312c004470 comp rx=0 tx=0).stop 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 shutdown_connections 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6810/3114250568,v1:172.21.15.77:6811/3114250568] conn(0x7f312806ef70 0x7f3128071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31280692a0 0x7f312806b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.892 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31380fdfe0 0x7f31380fbea0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 --2- 172.21.15.77:0/3986763576 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31380fe8f0 0x7f31380fc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:31.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 >> 172.21.15.77:0/3986763576 conn(0x7f3138071d90 msgr2=0x7f3138101840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:31.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 shutdown_connections 2024-02-19T18:57:31.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:31.874+0000 7f313eae4700 1 -- 172.21.15.77:0/3986763576 wait complete. 2024-02-19T18:57:32.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.459+0000 7ff831268700 1 -- 172.21.15.77:0/878268365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c102800 msgr2=0x7ff82c0fbe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:32.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.459+0000 7ff831268700 1 --2- 172.21.15.77:0/878268365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c102800 0x7ff82c0fbe70 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff820004940 tx=0x7ff820009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:32.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.459+0000 7ff831268700 1 -- 172.21.15.77:0/878268365 shutdown_connections 2024-02-19T18:57:32.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.459+0000 7ff831268700 1 --2- 172.21.15.77:0/878268365 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c102800 0x7ff82c0fbe70 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.459+0000 7ff831268700 1 --2- 172.21.15.77:0/878268365 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff82c101e60 0x7ff82c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.459+0000 7ff831268700 1 -- 172.21.15.77:0/878268365 >> 172.21.15.77:0/878268365 conn(0x7ff82c0f7a10 msgr2=0x7ff82c0f9e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:32.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.459+0000 7ff831268700 1 -- 172.21.15.77:0/878268365 shutdown_connections 2024-02-19T18:57:32.480 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.459+0000 7ff831268700 1 -- 172.21.15.77:0/878268365 wait complete. 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff831268700 1 Processor -- start 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff831268700 1 -- start start 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff831268700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c101e60 0x7ff82c113a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff831268700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff82c102800 0x7ff82c113f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff831268700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c1145e0 con 0x7ff82c101e60 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff831268700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff82c10d910 con 0x7ff82c102800 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff82affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c101e60 0x7ff82c113a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff82affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c101e60 0x7ff82c113a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47856/0 (socket says 172.21.15.77:47856) 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff82affd700 1 -- 172.21.15.77:0/3431402919 learned_addr learned my addr 172.21.15.77:0/3431402919 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff82a7fc700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff82c102800 0x7ff82c113f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff82affd700 1 -- 172.21.15.77:0/3431402919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff82c102800 msgr2=0x7ff82c113f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff82affd700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff82c102800 0x7ff82c113f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.460+0000 7ff82affd700 1 -- 172.21.15.77:0/3431402919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff820005040 con 0x7ff82c101e60 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.461+0000 7ff82affd700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c101e60 0x7ff82c113a50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff81c002a80 tx=0x7ff81c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:32.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.461+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81c007cc0 con 0x7ff82c101e60 2024-02-19T18:57:32.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.461+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff82c10dbf0 con 0x7ff82c101e60 2024-02-19T18:57:32.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.461+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff81c00f040 con 0x7ff82c101e60 2024-02-19T18:57:32.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.461+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff81c0096f0 con 0x7ff82c101e60 2024-02-19T18:57:32.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.461+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff82c10e130 con 0x7ff82c101e60 2024-02-19T18:57:32.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.462+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff82c10e610 con 0x7ff82c101e60 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.463+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff81c00d690 con 0x7ff82c101e60 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.463+0000 7ff8137fe700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8140692a0 0x7ff81406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.463+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff81c086610 con 0x7ff82c101e60 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.463+0000 7ff82a7fc700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8140692a0 0x7ff81406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.463+0000 7ff8137fe700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] conn(0x7ff81406ef70 0x7ff814071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.463+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 --> [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff814071a30 con 0x7ff81406ef70 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.463+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff81c088c00 con 0x7ff82c101e60 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.464+0000 7ff82b7fe700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] conn(0x7ff81406ef70 0x7ff814071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.464+0000 7ff82a7fc700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8140692a0 0x7ff81406b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff82c10f280 tx=0x7ff820009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.464+0000 7ff82b7fe700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] conn(0x7ff81406ef70 0x7ff814071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:32.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.465+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 <== osd.2 v2:172.21.15.77:6818/100949402 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7ff814071a30 con 0x7ff81406ef70 2024-02-19T18:57:32.487 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:32.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.490+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 --> [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff82c0fb610 con 0x7ff81406ef70 2024-02-19T18:57:32.508 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.490+0000 7ff8137fe700 1 -- 172.21.15.77:0/3431402919 <== osd.2 v2:172.21.15.77:6818/100949402 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff82c0fb610 con 0x7ff81406ef70 2024-02-19T18:57:32.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] conn(0x7ff81406ef70 msgr2=0x7ff814071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:32.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] conn(0x7ff81406ef70 0x7ff814071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8140692a0 msgr2=0x7ff81406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:32.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8140692a0 0x7ff81406b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff82c10f280 tx=0x7ff820009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:32.509 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c101e60 msgr2=0x7ff82c113a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c101e60 0x7ff82c113a50 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff81c002a80 tx=0x7ff81c0046b0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 shutdown_connections 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8140692a0 0x7ff81406b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff82c101e60 0x7ff82c113a50 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.77:6818/100949402,v1:172.21.15.77:6819/100949402] conn(0x7ff81406ef70 0x7ff814071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 --2- 172.21.15.77:0/3431402919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff82c102800 0x7ff82c113f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 >> 172.21.15.77:0/3431402919 conn(0x7ff82c0f7a10 msgr2=0x7ff82c066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 shutdown_connections 2024-02-19T18:57:32.510 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:32.491+0000 7ff831268700 1 -- 172.21.15.77:0/3431402919 wait complete. 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.298+0000 7f419e59d700 1 -- 172.21.15.77:0/999753666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 msgr2=0x7f4198103320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.298+0000 7f419e59d700 1 --2- 172.21.15.77:0/999753666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 0x7f4198103320 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4188004940 tx=0x7f4188009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 -- 172.21.15.77:0/999753666 shutdown_connections 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 --2- 172.21.15.77:0/999753666 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41981038f0 0x7f41980fcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 --2- 172.21.15.77:0/999753666 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 0x7f4198103320 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 -- 172.21.15.77:0/999753666 >> 172.21.15.77:0/999753666 conn(0x7f41980f8ae0 msgr2=0x7f41980faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 -- 172.21.15.77:0/999753666 shutdown_connections 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 -- 172.21.15.77:0/999753666 wait complete. 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 Processor -- start 2024-02-19T18:57:33.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 -- start start 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 0x7f4198070290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41981038f0 0x7f41980707d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4198070dd0 con 0x7f4198102f50 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.299+0000 7f419e59d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4198070f40 con 0x7f41981038f0 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4197fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 0x7f4198070290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4197fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 0x7f4198070290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47874/0 (socket says 172.21.15.77:47874) 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4197fff700 1 -- 172.21.15.77:0/242982919 learned_addr learned my addr 172.21.15.77:0/242982919 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f41977fe700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41981038f0 0x7f41980707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4197fff700 1 -- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41981038f0 msgr2=0x7f41980707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4197fff700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41981038f0 0x7f41980707d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4197fff700 1 -- 172.21.15.77:0/242982919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4188005040 con 0x7f4198102f50 2024-02-19T18:57:33.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4197fff700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 0x7f4198070290 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4188009100 tx=0x7f4188009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f418800e040 con 0x7f4198102f50 2024-02-19T18:57:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4198119ca0 con 0x7f4198102f50 2024-02-19T18:57:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4188013420 con 0x7f4198102f50 2024-02-19T18:57:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f418800bee0 con 0x7f4198102f50 2024-02-19T18:57:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.300+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419811a100 con 0x7f4198102f50 2024-02-19T18:57:33.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.301+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f419805e690 con 0x7f4198102f50 2024-02-19T18:57:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f418800ba10 con 0x7f4198102f50 2024-02-19T18:57:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f4194ff9700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f41800692a0 0x7f418006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4188088ba0 con 0x7f4198102f50 2024-02-19T18:57:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f41977fe700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f41800692a0 0x7f418006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f4194ff9700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] conn(0x7f418006ef70 0x7f4180071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 --> [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4180071a30 con 0x7f418006ef70 2024-02-19T18:57:33.321 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4188088f80 con 0x7f4198102f50 2024-02-19T18:57:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f41977fe700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f41800692a0 0x7f418006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4198071c90 tx=0x7f418c00b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.302+0000 7f419cb3a700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] conn(0x7f418006ef70 0x7f4180071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.303+0000 7f419cb3a700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] conn(0x7f418006ef70 0x7f4180071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:33.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.304+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 <== osd.5 v2:172.21.15.104:6816/2114623660 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4180071a30 con 0x7f418006ef70 2024-02-19T18:57:33.336 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.318+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 --> [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f419805ea90 con 0x7f418006ef70 2024-02-19T18:57:33.337 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.319+0000 7f4194ff9700 1 -- 172.21.15.77:0/242982919 <== osd.5 v2:172.21.15.104:6816/2114623660 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f419805ea90 con 0x7f418006ef70 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.319+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] conn(0x7f418006ef70 msgr2=0x7f4180071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.319+0000 7f419e59d700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] conn(0x7f418006ef70 0x7f4180071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f41800692a0 msgr2=0x7f418006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f41800692a0 0x7f418006b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f4198071c90 tx=0x7f418c00b040 comp rx=0 tx=0).stop 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 msgr2=0x7f4198070290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 0x7f4198070290 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4188009100 tx=0x7f4188009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 shutdown_connections 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f41800692a0 0x7f418006b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4198102f50 0x7f4198070290 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:6816/2114623660,v1:172.21.15.104:6817/2114623660] conn(0x7f418006ef70 0x7f4180071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 --2- 172.21.15.77:0/242982919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f41981038f0 0x7f41980707d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:33.338 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 >> 172.21.15.77:0/242982919 conn(0x7f41980f8ae0 msgr2=0x7f41980fa4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:33.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 shutdown_connections 2024-02-19T18:57:33.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:33.320+0000 7f419e59d700 1 -- 172.21.15.77:0/242982919 wait complete. 2024-02-19T18:57:33.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:33 smithi077 ceph-mon[29247]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:33.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:33 smithi104 ceph-mon[42320]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:34.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:34.366 INFO:teuthology.orchestra.run.smithi077.stdout:81604378648 2024-02-19T18:57:34.366 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-19T18:57:34.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:34.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.944+0000 7fe5f1704700 1 -- 172.21.15.77:0/3428092964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fc630 msgr2=0x7fe5ec109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:34.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.944+0000 7fe5f1704700 1 --2- 172.21.15.77:0/3428092964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fc630 0x7fe5ec109390 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fe5e0009ce0 tx=0x7fe5e0004d10 comp rx=0 tx=0).stop 2024-02-19T18:57:34.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.944+0000 7fe5f1704700 1 -- 172.21.15.77:0/3428092964 shutdown_connections 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.944+0000 7fe5f1704700 1 --2- 172.21.15.77:0/3428092964 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fc630 0x7fe5ec109390 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.944+0000 7fe5f1704700 1 --2- 172.21.15.77:0/3428092964 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec0fbd20 0x7fe5ec0fc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.944+0000 7fe5f1704700 1 -- 172.21.15.77:0/3428092964 >> 172.21.15.77:0/3428092964 conn(0x7fe5ec0f79d0 msgr2=0x7fe5ec0f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.944+0000 7fe5f1704700 1 -- 172.21.15.77:0/3428092964 shutdown_connections 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.944+0000 7fe5f1704700 1 -- 172.21.15.77:0/3428092964 wait complete. 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5f1704700 1 Processor -- start 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5f1704700 1 -- start start 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5f1704700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fbd20 0x7fe5ec115e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5f1704700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec0fc630 0x7fe5ec1163c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5f1704700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec116a10 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5f1704700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec10fd20 con 0x7fe5ec0fc630 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5eaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fbd20 0x7fe5ec115e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5eaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fbd20 0x7fe5ec115e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:47896/0 (socket says 172.21.15.77:47896) 2024-02-19T18:57:34.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5eaffd700 1 -- 172.21.15.77:0/94735200 learned_addr learned my addr 172.21.15.77:0/94735200 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5ea7fc700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec0fc630 0x7fe5ec1163c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5eaffd700 1 -- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec0fc630 msgr2=0x7fe5ec1163c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5eaffd700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec0fc630 0x7fe5ec1163c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5eaffd700 1 -- 172.21.15.77:0/94735200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e0009940 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.945+0000 7fe5eaffd700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fbd20 0x7fe5ec115e80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400c830 tx=0x7fe5e4004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.946+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5e400a760 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.946+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5e400ad90 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.946+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec110000 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.946+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5e40113a0 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.946+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec110540 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.946+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe5ec110a80 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.947+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe5e4012400 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.947+0000 7fe5d37fe700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5d40692a0 0x7fe5d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.947+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe5e40878d0 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.947+0000 7fe5ea7fc700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5d40692a0 0x7fe5d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.948+0000 7fe5d37fe700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] conn(0x7fe5d406ef70 0x7fe5d4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.948+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 --> [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe5d4071a30 con 0x7fe5d406ef70 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.948+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5e4087cb0 con 0x7fe5ec0fbd20 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.948+0000 7fe5eb7fe700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] conn(0x7fe5d406ef70 0x7fe5d4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.948+0000 7fe5ea7fc700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5d40692a0 0x7fe5d406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000b740 tx=0x7fe5e0004d10 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.948+0000 7fe5eb7fe700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] conn(0x7fe5d406ef70 0x7fe5d4071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:34.967 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.949+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 <== osd.4 v2:172.21.15.104:6808/3596569551 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fe5d4071a30 con 0x7fe5d406ef70 2024-02-19T18:57:34.981 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.964+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 --> [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe5ec05e690 con 0x7fe5d406ef70 2024-02-19T18:57:34.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.964+0000 7fe5d37fe700 1 -- 172.21.15.77:0/94735200 <== osd.4 v2:172.21.15.104:6808/3596569551 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe5ec05e690 con 0x7fe5d406ef70 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.964+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] conn(0x7fe5d406ef70 msgr2=0x7fe5d4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.964+0000 7fe5f1704700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] conn(0x7fe5d406ef70 0x7fe5d4071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5d40692a0 msgr2=0x7fe5d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5d40692a0 0x7fe5d406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe5e000b740 tx=0x7fe5e0004d10 comp rx=0 tx=0).stop 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fbd20 msgr2=0x7fe5ec115e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fbd20 0x7fe5ec115e80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe5e400c830 tx=0x7fe5e4004470 comp rx=0 tx=0).stop 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 shutdown_connections 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5d40692a0 0x7fe5d406b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:6808/3596569551,v1:172.21.15.104:6809/3596569551] conn(0x7fe5d406ef70 0x7fe5d4071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5ec0fbd20 0x7fe5ec115e80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:34.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 --2- 172.21.15.77:0/94735200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5ec0fc630 0x7fe5ec1163c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 >> 172.21.15.77:0/94735200 conn(0x7fe5ec0f79d0 msgr2=0x7fe5ec0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 shutdown_connections 2024-02-19T18:57:34.984 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:34.965+0000 7fe5f1704700 1 -- 172.21.15.77:0/94735200 wait complete. 2024-02-19T18:57:35.487 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:35 smithi077 ceph-mon[29247]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:35.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:35 smithi104 ceph-mon[42320]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:35.644 INFO:teuthology.orchestra.run.smithi077.stdout:150323855373 2024-02-19T18:57:35.644 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-19T18:57:36.777 INFO:teuthology.orchestra.run.smithi077.stdout:171798691850 2024-02-19T18:57:36.777 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-19T18:57:37.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:37 smithi077 ceph-mon[29247]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:37.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:37 smithi104 ceph-mon[42320]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:37.805 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:37.815 INFO:teuthology.orchestra.run.smithi077.stdout:60129542174 2024-02-19T18:57:37.815 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-19T18:57:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 -- 172.21.15.77:0/2017451545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a00fb570 msgr2=0x7f57a00fb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2017451545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a00fb570 0x7f57a00fb9c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5790004940 tx=0x7f5790009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 -- 172.21.15.77:0/2017451545 shutdown_connections 2024-02-19T18:57:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2017451545 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a00fb570 0x7f57a00fb9c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2017451545 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a0103140 0x7f57a0103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 -- 172.21.15.77:0/2017451545 >> 172.21.15.77:0/2017451545 conn(0x7f57a0071560 msgr2=0x7f57a0071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 -- 172.21.15.77:0/2017451545 shutdown_connections 2024-02-19T18:57:38.313 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 -- 172.21.15.77:0/2017451545 wait complete. 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 Processor -- start 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.294+0000 7f57a6d75700 1 -- start start 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57a6d75700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a00fb570 0x7f57a00fc750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57a6d75700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a0103140 0x7f57a00fcc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57a6d75700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00fd320 con 0x7f57a0103140 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57a6d75700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a00fd490 con 0x7f57a00fb570 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f5797fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a0103140 0x7f57a00fcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f5797fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a0103140 0x7f57a00fcc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:32984/0 (socket says 172.21.15.77:32984) 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f5797fff700 1 -- 172.21.15.77:0/2061966223 learned_addr learned my addr 172.21.15.77:0/2061966223 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57a4b11700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a00fb570 0x7f57a00fc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f5797fff700 1 -- 172.21.15.77:0/2061966223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a00fb570 msgr2=0x7f57a00fc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f5797fff700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a00fb570 0x7f57a00fc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f5797fff700 1 -- 172.21.15.77:0/2061966223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5790005040 con 0x7f57a0103140 2024-02-19T18:57:38.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57a4b11700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a00fb570 0x7f57a00fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:57:38.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f5797fff700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a0103140 0x7f57a00fcc90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f579000b3b0 tx=0x7f5790009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:38.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f579000e040 con 0x7f57a0103140 2024-02-19T18:57:38.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a0119e60 con 0x7f57a0103140 2024-02-19T18:57:38.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f579000bbf0 con 0x7f57a0103140 2024-02-19T18:57:38.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.295+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f579001c630 con 0x7f57a0103140 2024-02-19T18:57:38.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.296+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a011a320 con 0x7f57a0103140 2024-02-19T18:57:38.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.296+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f57a005e650 con 0x7f57a0103140 2024-02-19T18:57:38.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.297+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f5790013420 con 0x7f57a0103140 2024-02-19T18:57:38.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.297+0000 7f57957fa700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f57880692a0 0x7f578806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:38.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.297+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5790088210 con 0x7f57a0103140 2024-02-19T18:57:38.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.297+0000 7f57a4b11700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f57880692a0 0x7f578806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:38.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.297+0000 7f57957fa700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] conn(0x7f578806ef70 0x7f5788071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:38.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.297+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 --> [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5788071a30 con 0x7f578806ef70 2024-02-19T18:57:38.316 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.297+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f579008a860 con 0x7f57a0103140 2024-02-19T18:57:38.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.298+0000 7f57a5312700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] conn(0x7f578806ef70 0x7f5788071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:38.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.298+0000 7f57a4b11700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f57880692a0 0x7f578806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f579c006fd0 tx=0x7f579c008040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:38.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.298+0000 7f57a5312700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] conn(0x7f578806ef70 0x7f5788071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:38.317 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.298+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 <== osd.0 v2:172.21.15.77:6802/806127290 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5788071a30 con 0x7f578806ef70 2024-02-19T18:57:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.313+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 --> [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f57a005ea00 con 0x7f578806ef70 2024-02-19T18:57:38.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.313+0000 7f57957fa700 1 -- 172.21.15.77:0/2061966223 <== osd.0 v2:172.21.15.77:6802/806127290 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f57a005ea00 con 0x7f578806ef70 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] conn(0x7f578806ef70 msgr2=0x7f5788071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] conn(0x7f578806ef70 0x7f5788071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f57880692a0 msgr2=0x7f578806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f57880692a0 0x7f578806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f579c006fd0 tx=0x7f579c008040 comp rx=0 tx=0).stop 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a0103140 msgr2=0x7f57a00fcc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a0103140 0x7f57a00fcc90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f579000b3b0 tx=0x7f5790009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 shutdown_connections 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f57880692a0 0x7f578806b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:6802/806127290,v1:172.21.15.77:6803/806127290] conn(0x7f578806ef70 0x7f5788071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f57a00fb570 0x7f57a00fc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 --2- 172.21.15.77:0/2061966223 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f57a0103140 0x7f57a00fcc90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:38.332 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 >> 172.21.15.77:0/2061966223 conn(0x7f57a0071560 msgr2=0x7f57a00f96a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:38.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 shutdown_connections 2024-02-19T18:57:38.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:38.314+0000 7f57a6d75700 1 -- 172.21.15.77:0/2061966223 wait complete. 2024-02-19T18:57:38.933 INFO:teuthology.orchestra.run.smithi077.stdout:38654705701 2024-02-19T18:57:38.934 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-19T18:57:39.159 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:38 smithi077 ceph-mon[29247]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:39.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:38 smithi104 ceph-mon[42320]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:39.610 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:39.611 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:39.611 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:40.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 -- 172.21.15.77:0/3834852454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c101e40 msgr2=0x7fe12c102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 --2- 172.21.15.77:0/3834852454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c101e40 0x7fe12c102210 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe124009ce0 tx=0x7fe124004d10 comp rx=0 tx=0).stop 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 -- 172.21.15.77:0/3834852454 shutdown_connections 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 --2- 172.21.15.77:0/3834852454 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe12c1027e0 0x7fe12c0fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 --2- 172.21.15.77:0/3834852454 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c101e40 0x7fe12c102210 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 -- 172.21.15.77:0/3834852454 >> 172.21.15.77:0/3834852454 conn(0x7fe12c071d90 msgr2=0x7fe12c072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 -- 172.21.15.77:0/3834852454 shutdown_connections 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 -- 172.21.15.77:0/3834852454 wait complete. 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.524+0000 7fe131dcd700 1 Processor -- start 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe131dcd700 1 -- start start 2024-02-19T18:57:40.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe131dcd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe12c101e40 0x7fe12c115df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe131dcd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c1027e0 0x7fe12c116330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe131dcd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c116a10 con 0x7fe12c1027e0 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe131dcd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c10fc90 con 0x7fe12c101e40 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe12affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c1027e0 0x7fe12c116330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe12affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c1027e0 0x7fe12c116330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:33008/0 (socket says 172.21.15.77:33008) 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe12affd700 1 -- 172.21.15.77:0/1297626286 learned_addr learned my addr 172.21.15.77:0/1297626286 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe12b7fe700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe12c101e40 0x7fe12c115df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe12affd700 1 -- 172.21.15.77:0/1297626286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe12c101e40 msgr2=0x7fe12c115df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe12affd700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe12c101e40 0x7fe12c115df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe12affd700 1 -- 172.21.15.77:0/1297626286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe124009940 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe12affd700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c1027e0 0x7fe12c116330 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe12000b790 tx=0x7fe120009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.525+0000 7fe113fff700 1 -- 172.21.15.77:0/1297626286 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe12000f6f0 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.526+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12c10ff10 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.526+0000 7fe113fff700 1 -- 172.21.15.77:0/1297626286 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe12000fd20 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.526+0000 7fe113fff700 1 -- 172.21.15.77:0/1297626286 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1200103a0 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.526+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12c110450 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.526+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12c0738b0 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.527+0000 7fe113fff700 1 -- 172.21.15.77:0/1297626286 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe120011400 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.527+0000 7fe113fff700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1140692a0 0x7fe11406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.527+0000 7fe113fff700 1 -- 172.21.15.77:0/1297626286 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe120086d10 con 0x7fe12c1027e0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.527+0000 7fe12b7fe700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1140692a0 0x7fe11406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:40.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.528+0000 7fe12b7fe700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1140692a0 0x7fe11406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe12400b740 tx=0x7fe124004d10 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:40.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.532+0000 7fe113fff700 1 -- 172.21.15.77:0/1297626286 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe120056d90 con 0x7fe12c1027e0 2024-02-19T18:57:40.582 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:40.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.724+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe12c111400 con 0x7fe12c1027e0 2024-02-19T18:57:40.742 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.725+0000 7fe113fff700 1 -- 172.21.15.77:0/1297626286 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe120056920 con 0x7fe12c1027e0 2024-02-19T18:57:40.742 INFO:teuthology.orchestra.run.smithi077.stdout:115964117012 2024-02-19T18:57:40.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1140692a0 msgr2=0x7fe11406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:40.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1140692a0 0x7fe11406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe12400b740 tx=0x7fe124004d10 comp rx=0 tx=0).stop 2024-02-19T18:57:40.744 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c1027e0 msgr2=0x7fe12c116330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c1027e0 0x7fe12c116330 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe12000b790 tx=0x7fe120009220 comp rx=0 tx=0).stop 2024-02-19T18:57:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 shutdown_connections 2024-02-19T18:57:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe1140692a0 0x7fe11406b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe12c101e40 0x7fe12c115df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 --2- 172.21.15.77:0/1297626286 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe12c1027e0 0x7fe12c116330 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 >> 172.21.15.77:0/1297626286 conn(0x7fe12c071d90 msgr2=0x7fe12c0656c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 shutdown_connections 2024-02-19T18:57:40.745 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:40.727+0000 7fe131dcd700 1 -- 172.21.15.77:0/1297626286 wait complete. 2024-02-19T18:57:41.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:41 smithi077 ceph-mon[29247]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:41.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:41 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1297626286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T18:57:41.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:41 smithi104 ceph-mon[42320]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:41.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:41 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1297626286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-19T18:57:43.474 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:43 smithi077 ceph-mon[29247]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:43.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:43 smithi104 ceph-mon[42320]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:43.785 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:44.220 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117010 got 115964117012 for osd.3 2024-02-19T18:57:44.220 DEBUG:teuthology.parallel:result is None 2024-02-19T18:57:45.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:45 smithi077 ceph-mon[29247]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:45.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:45 smithi104 ceph-mon[42320]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:46.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.538+0000 7f6be7638700 1 -- 172.21.15.77:0/41734443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be0101e50 msgr2=0x7f6be0102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:46.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.538+0000 7f6be7638700 1 --2- 172.21.15.77:0/41734443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be0101e50 0x7f6be0102220 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc004940 tx=0x7f6bdc009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:46.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.538+0000 7f6be7638700 1 -- 172.21.15.77:0/41734443 shutdown_connections 2024-02-19T18:57:46.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.538+0000 7f6be7638700 1 --2- 172.21.15.77:0/41734443 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6be01027f0 0x7f6be00fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.538+0000 7f6be7638700 1 --2- 172.21.15.77:0/41734443 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be0101e50 0x7f6be0102220 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.538+0000 7f6be7638700 1 -- 172.21.15.77:0/41734443 >> 172.21.15.77:0/41734443 conn(0x7f6be00f79e0 msgr2=0x7f6be00f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:46.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.538+0000 7f6be7638700 1 -- 172.21.15.77:0/41734443 shutdown_connections 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.538+0000 7f6be7638700 1 -- 172.21.15.77:0/41734443 wait complete. 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be7638700 1 Processor -- start 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be7638700 1 -- start start 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be7638700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6be0101e50 0x7f6be0115cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be7638700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be01027f0 0x7f6be01161f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be7638700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0116840 con 0x7f6be01027f0 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be7638700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be010fb50 con 0x7f6be0101e50 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be4bd3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be01027f0 0x7f6be01161f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be4bd3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be01027f0 0x7f6be01161f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:53994/0 (socket says 172.21.15.77:53994) 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be4bd3700 1 -- 172.21.15.77:0/1585776245 learned_addr learned my addr 172.21.15.77:0/1585776245 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be53d4700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6be0101e50 0x7f6be0115cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be4bd3700 1 -- 172.21.15.77:0/1585776245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6be0101e50 msgr2=0x7f6be0115cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be4bd3700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6be0101e50 0x7f6be0115cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be4bd3700 1 -- 172.21.15.77:0/1585776245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc005040 con 0x7f6be01027f0 2024-02-19T18:57:46.562 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.539+0000 7f6be4bd3700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be01027f0 0x7f6be01161f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000c890 tx=0x7f6bd00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.540+0000 7f6bd5ffb700 1 -- 172.21.15.77:0/1585776245 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bd000a700 con 0x7f6be01027f0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.540+0000 7f6bd5ffb700 1 -- 172.21.15.77:0/1585776245 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bd000ad30 con 0x7f6be01027f0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.540+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be010fe30 con 0x7f6be01027f0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.540+0000 7f6bd5ffb700 1 -- 172.21.15.77:0/1585776245 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bd00113a0 con 0x7f6be01027f0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.540+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be0110370 con 0x7f6be01027f0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.541+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be0064b00 con 0x7f6be01027f0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.541+0000 7f6bd5ffb700 1 -- 172.21.15.77:0/1585776245 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6bd0012400 con 0x7f6be01027f0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.541+0000 7f6bd5ffb700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6bcc0692a0 0x7f6bcc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.542+0000 7f6bd5ffb700 1 -- 172.21.15.77:0/1585776245 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6bd0086f60 con 0x7f6be01027f0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.542+0000 7f6be53d4700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6bcc0692a0 0x7f6bcc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:46.563 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.542+0000 7f6be53d4700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6bcc0692a0 0x7f6bcc06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc004f90 tx=0x7f6bdc009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:46.564 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.546+0000 7f6bd5ffb700 1 -- 172.21.15.77:0/1585776245 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6bd0057000 con 0x7f6be01027f0 2024-02-19T18:57:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.737+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6be00fb5e0 con 0x7f6be01027f0 2024-02-19T18:57:46.755 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.737+0000 7f6bd5ffb700 1 -- 172.21.15.77:0/1585776245 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6bd0056b90 con 0x7f6be01027f0 2024-02-19T18:57:46.755 INFO:teuthology.orchestra.run.smithi077.stdout:81604378651 2024-02-19T18:57:46.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6bcc0692a0 msgr2=0x7f6bcc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6bcc0692a0 0x7f6bcc06b750 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc004f90 tx=0x7f6bdc009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be01027f0 msgr2=0x7f6be01161f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be01027f0 0x7f6be01161f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000c890 tx=0x7f6bd00044c0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 shutdown_connections 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6bcc0692a0 0x7f6bcc06b750 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6be0101e50 0x7f6be0115cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 --2- 172.21.15.77:0/1585776245 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6be01027f0 0x7f6be01161f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 >> 172.21.15.77:0/1585776245 conn(0x7f6be00f79e0 msgr2=0x7f6be0065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 shutdown_connections 2024-02-19T18:57:46.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.740+0000 7f6be7638700 1 -- 172.21.15.77:0/1585776245 wait complete. 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.848+0000 7fcf022d0700 1 -- 172.21.15.77:0/3636448310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 msgr2=0x7fcefc0fbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.848+0000 7fcf022d0700 1 --2- 172.21.15.77:0/3636448310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 0x7fcefc0fbea0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcef0004940 tx=0x7fcef0009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.848+0000 7fcf022d0700 1 -- 172.21.15.77:0/3636448310 shutdown_connections 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.848+0000 7fcf022d0700 1 --2- 172.21.15.77:0/3636448310 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 0x7fcefc0fbea0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.848+0000 7fcf022d0700 1 --2- 172.21.15.77:0/3636448310 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc101e40 0x7fcefc102210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.848+0000 7fcf022d0700 1 -- 172.21.15.77:0/3636448310 >> 172.21.15.77:0/3636448310 conn(0x7fcefc071d90 msgr2=0x7fcefc072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.848+0000 7fcf022d0700 1 -- 172.21.15.77:0/3636448310 shutdown_connections 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.848+0000 7fcf022d0700 1 -- 172.21.15.77:0/3636448310 wait complete. 2024-02-19T18:57:46.870 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcf022d0700 1 Processor -- start 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcf022d0700 1 -- start start 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcf022d0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc101e40 0x7fcefc10eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcf022d0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 0x7fcefc10f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcf022d0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc10f680 con 0x7fcefc1027e0 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcf022d0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcefc10f7f0 con 0x7fcefc101e40 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcefb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 0x7fcefc10f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcefb7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 0x7fcefc10f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54008/0 (socket says 172.21.15.77:54008) 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcefb7fe700 1 -- 172.21.15.77:0/4033600270 learned_addr learned my addr 172.21.15.77:0/4033600270 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.849+0000 7fcefbfff700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc101e40 0x7fcefc10eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcefb7fe700 1 -- 172.21.15.77:0/4033600270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc101e40 msgr2=0x7fcefc10eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcefb7fe700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc101e40 0x7fcefc10eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcefb7fe700 1 -- 172.21.15.77:0/4033600270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef0005040 con 0x7fcefc1027e0 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcefb7fe700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 0x7fcefc10f030 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009100 tx=0x7fcef0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcef8ff9700 1 -- 172.21.15.77:0/4033600270 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcef000e040 con 0x7fcefc1027e0 2024-02-19T18:57:46.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcefc117a60 con 0x7fcefc1027e0 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcef8ff9700 1 -- 172.21.15.77:0/4033600270 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcef000bbf0 con 0x7fcefc1027e0 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcef8ff9700 1 -- 172.21.15.77:0/4033600270 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcef001c3f0 con 0x7fcefc1027e0 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.850+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcefc117fa0 con 0x7fcefc1027e0 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.851+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcefc104310 con 0x7fcefc1027e0 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.851+0000 7fcef8ff9700 1 -- 172.21.15.77:0/4033600270 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fcef000bd60 con 0x7fcefc1027e0 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.852+0000 7fcef8ff9700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcee40692a0 0x7fcee406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.852+0000 7fcef8ff9700 1 -- 172.21.15.77:0/4033600270 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcef0088120 con 0x7fcefc1027e0 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.852+0000 7fcefbfff700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcee40692a0 0x7fcee406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:46.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.852+0000 7fcefbfff700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcee40692a0 0x7fcee406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fceec004eb0 tx=0x7fceec00b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:46.874 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:46.856+0000 7fcef8ff9700 1 -- 172.21.15.77:0/4033600270 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fcef00546f0 con 0x7fcefc1027e0 2024-02-19T18:57:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.041+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fcefc065ee0 con 0x7fcefc1027e0 2024-02-19T18:57:47.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.042+0000 7fcef8ff9700 1 -- 172.21.15.77:0/4033600270 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcef0057d10 con 0x7fcefc1027e0 2024-02-19T18:57:47.060 INFO:teuthology.orchestra.run.smithi077.stdout:150323855376 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcee40692a0 msgr2=0x7fcee406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcee40692a0 0x7fcee406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fceec004eb0 tx=0x7fceec00b040 comp rx=0 tx=0).stop 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 msgr2=0x7fcefc10f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 0x7fcefc10f030 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fcef0009100 tx=0x7fcef0009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 shutdown_connections 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fcee40692a0 0x7fcee406b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fcefc101e40 0x7fcefc10eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 --2- 172.21.15.77:0/4033600270 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fcefc1027e0 0x7fcefc10f030 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 >> 172.21.15.77:0/4033600270 conn(0x7fcefc071d90 msgr2=0x7fcefc0fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 shutdown_connections 2024-02-19T18:57:47.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:47.044+0000 7fcf022d0700 1 -- 172.21.15.77:0/4033600270 wait complete. 2024-02-19T18:57:47.490 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:47 smithi077 ceph-mon[29247]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:47.490 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:47 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1585776245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T18:57:47.490 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:47 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4033600270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T18:57:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:47 smithi104 ceph-mon[42320]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:47.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:47 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1585776245' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-19T18:57:47.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:47 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/4033600270' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-19T18:57:47.816 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855373 got 150323855376 for osd.4 2024-02-19T18:57:47.817 DEBUG:teuthology.parallel:result is None 2024-02-19T18:57:48.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.805+0000 7f6c8eda6700 1 -- 172.21.15.77:0/97204909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 msgr2=0x7f6c880fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:48.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.805+0000 7f6c8eda6700 1 --2- 172.21.15.77:0/97204909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 0x7f6c880fcf60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6c74004940 tx=0x7f6c74009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:48.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.805+0000 7f6c8eda6700 1 -- 172.21.15.77:0/97204909 shutdown_connections 2024-02-19T18:57:48.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.805+0000 7f6c8eda6700 1 --2- 172.21.15.77:0/97204909 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 0x7f6c880fcf60 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:48.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.805+0000 7f6c8eda6700 1 --2- 172.21.15.77:0/97204909 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c88102f50 0x7f6c88103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:48.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.805+0000 7f6c8eda6700 1 -- 172.21.15.77:0/97204909 >> 172.21.15.77:0/97204909 conn(0x7f6c880f8ae0 msgr2=0x7f6c880faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.805+0000 7f6c8eda6700 1 -- 172.21.15.77:0/97204909 shutdown_connections 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.805+0000 7f6c8eda6700 1 -- 172.21.15.77:0/97204909 wait complete. 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c8eda6700 1 Processor -- start 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c8eda6700 1 -- start start 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c8eda6700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c88102f50 0x7f6c88113ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c8eda6700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 0x7f6c88113ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c8eda6700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c88114640 con 0x7f6c881038f0 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c8eda6700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8810d970 con 0x7f6c88102f50 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c87fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 0x7f6c88113ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c87fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 0x7f6c88113ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54024/0 (socket says 172.21.15.77:54024) 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c87fff700 1 -- 172.21.15.77:0/3781750273 learned_addr learned my addr 172.21.15.77:0/3781750273 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.806+0000 7f6c8cb42700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c88102f50 0x7f6c88113ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:48.825 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c87fff700 1 -- 172.21.15.77:0/3781750273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c88102f50 msgr2=0x7f6c88113ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c87fff700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c88102f50 0x7f6c88113ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c87fff700 1 -- 172.21.15.77:0/3781750273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74005040 con 0x7f6c881038f0 2024-02-19T18:57:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c87fff700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 0x7f6c88113ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009100 tx=0x7f6c74009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c857fa700 1 -- 172.21.15.77:0/3781750273 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c7400e040 con 0x7f6c881038f0 2024-02-19T18:57:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c857fa700 1 -- 172.21.15.77:0/3781750273 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c74004290 con 0x7f6c881038f0 2024-02-19T18:57:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8810dbf0 con 0x7f6c881038f0 2024-02-19T18:57:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c857fa700 1 -- 172.21.15.77:0/3781750273 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c7401c3f0 con 0x7f6c881038f0 2024-02-19T18:57:48.826 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.807+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8810e0e0 con 0x7f6c881038f0 2024-02-19T18:57:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.808+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c88105420 con 0x7f6c881038f0 2024-02-19T18:57:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.808+0000 7f6c857fa700 1 -- 172.21.15.77:0/3781750273 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6c7400bbf0 con 0x7f6c881038f0 2024-02-19T18:57:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.809+0000 7f6c857fa700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c780692b0 0x7f6c7806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.809+0000 7f6c857fa700 1 -- 172.21.15.77:0/3781750273 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6c740880b0 con 0x7f6c881038f0 2024-02-19T18:57:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.809+0000 7f6c8cb42700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c780692b0 0x7f6c7806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:48.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.809+0000 7f6c8cb42700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c780692b0 0x7f6c7806b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c004f60 tx=0x7f6c7c00b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:48.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.813+0000 7f6c857fa700 1 -- 172.21.15.77:0/3781750273 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6c740581b0 con 0x7f6c881038f0 2024-02-19T18:57:49.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.990+0000 7fa9c871c700 1 -- 172.21.15.77:0/1657703456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 msgr2=0x7fa9c0102210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:49.007 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.990+0000 7fa9c871c700 1 --2- 172.21.15.77:0/1657703456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 0x7fa9c0102210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc004940 tx=0x7fa9bc009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.990+0000 7fa9c871c700 1 -- 172.21.15.77:0/1657703456 shutdown_connections 2024-02-19T18:57:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.990+0000 7fa9c871c700 1 --2- 172.21.15.77:0/1657703456 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c01027e0 0x7fa9c00fbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.990+0000 7fa9c871c700 1 --2- 172.21.15.77:0/1657703456 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 0x7fa9c0102210 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.990+0000 7fa9c871c700 1 -- 172.21.15.77:0/1657703456 >> 172.21.15.77:0/1657703456 conn(0x7fa9c0071d90 msgr2=0x7fa9c0072190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.991+0000 7fa9c871c700 1 -- 172.21.15.77:0/1657703456 shutdown_connections 2024-02-19T18:57:49.008 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.991+0000 7fa9c871c700 1 -- 172.21.15.77:0/1657703456 wait complete. 2024-02-19T18:57:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.991+0000 7fa9c871c700 1 Processor -- start 2024-02-19T18:57:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.991+0000 7fa9c871c700 1 -- start start 2024-02-19T18:57:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.991+0000 7fa9c871c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 0x7fa9c010eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.991+0000 7fa9c871c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c01027e0 0x7fa9c010f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.991+0000 7fa9c871c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c010f5b0 con 0x7fa9c0101e40 2024-02-19T18:57:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.991+0000 7fa9c871c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c010f6f0 con 0x7fa9c01027e0 2024-02-19T18:57:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c64b8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 0x7fa9c010eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:49.010 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c64b8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 0x7fa9c010eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54034/0 (socket says 172.21.15.77:54034) 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c64b8700 1 -- 172.21.15.77:0/3513693299 learned_addr learned my addr 172.21.15.77:0/3513693299 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c5cb7700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c01027e0 0x7fa9c010f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c64b8700 1 -- 172.21.15.77:0/3513693299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c01027e0 msgr2=0x7fa9c010f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c64b8700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c01027e0 0x7fa9c010f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c64b8700 1 -- 172.21.15.77:0/3513693299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9bc005040 con 0x7fa9c0101e40 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c64b8700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 0x7fa9c010eae0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009100 tx=0x7fa9bc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/3513693299 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9bc00e040 con 0x7fa9c0101e40 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/3513693299 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9bc013420 con 0x7fa9c0101e40 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c01179f0 con 0x7fa9c0101e40 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/3513693299 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9bc00bee0 con 0x7fa9c0101e40 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.992+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c0117f00 con 0x7fa9c0101e40 2024-02-19T18:57:49.011 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.993+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c00ff150 con 0x7fa9c0101e40 2024-02-19T18:57:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.994+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/3513693299 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa9bc00ba10 con 0x7fa9c0101e40 2024-02-19T18:57:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.994+0000 7fa9b6ffd700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa9ac0692a0 0x7fa9ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.994+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/3513693299 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa9bc088160 con 0x7fa9c0101e40 2024-02-19T18:57:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.994+0000 7fa9c5cb7700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa9ac0692a0 0x7fa9ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:49.012 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.995+0000 7fa9c5cb7700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa9ac0692a0 0x7fa9ac06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0066180 tx=0x7fa9b000b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:49.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:48.999+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/3513693299 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa9bc0547d0 con 0x7fa9c0101e40 2024-02-19T18:57:49.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.008+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f6c8810a050 con 0x7f6c881038f0 2024-02-19T18:57:49.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.008+0000 7f6c857fa700 1 -- 172.21.15.77:0/3781750273 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6c74057d40 con 0x7f6c881038f0 2024-02-19T18:57:49.025 INFO:teuthology.orchestra.run.smithi077.stdout:171798691852 2024-02-19T18:57:49.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c780692b0 msgr2=0x7f6c7806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:49.027 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c780692b0 0x7f6c7806b760 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c004f60 tx=0x7f6c7c00b040 comp rx=0 tx=0).stop 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 msgr2=0x7f6c88113ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 0x7f6c88113ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6c74009100 tx=0x7f6c74009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 shutdown_connections 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6c780692b0 0x7f6c7806b760 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6c88102f50 0x7f6c88113ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 --2- 172.21.15.77:0/3781750273 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6c881038f0 0x7f6c88113ff0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 >> 172.21.15.77:0/3781750273 conn(0x7f6c880f8ae0 msgr2=0x7f6c880fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.010+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 shutdown_connections 2024-02-19T18:57:49.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.011+0000 7f6c8eda6700 1 -- 172.21.15.77:0/3781750273 wait complete. 2024-02-19T18:57:49.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:48 smithi077 ceph-mon[29247]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:49.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.190+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa9c0065ec0 con 0x7fa9c0101e40 2024-02-19T18:57:49.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.191+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/3513693299 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa9bc057df0 con 0x7fa9c0101e40 2024-02-19T18:57:49.208 INFO:teuthology.orchestra.run.smithi077.stdout:38654705703 2024-02-19T18:57:49.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa9ac0692a0 msgr2=0x7fa9ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa9ac0692a0 0x7fa9ac06b750 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0066180 tx=0x7fa9b000b040 comp rx=0 tx=0).stop 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 msgr2=0x7fa9c010eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 0x7fa9c010eae0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009100 tx=0x7fa9bc009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 shutdown_connections 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa9ac0692a0 0x7fa9ac06b750 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c0101e40 0x7fa9c010eae0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 --2- 172.21.15.77:0/3513693299 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c01027e0 0x7fa9c010f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 >> 172.21.15.77:0/3513693299 conn(0x7fa9c0071d90 msgr2=0x7fa9c00fada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 shutdown_connections 2024-02-19T18:57:49.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:49.193+0000 7fa9c871c700 1 -- 172.21.15.77:0/3513693299 wait complete. 2024-02-19T18:57:49.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:48 smithi104 ceph-mon[42320]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:49.841 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705701 got 38654705703 for osd.0 2024-02-19T18:57:49.841 DEBUG:teuthology.parallel:result is None 2024-02-19T18:57:50.132 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:49 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3781750273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T18:57:50.133 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:50.133 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:49 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3513693299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.212+0000 7f446cf48700 1 -- 172.21.15.77:0/1473140348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 msgr2=0x7f4468102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.212+0000 7f446cf48700 1 --2- 172.21.15.77:0/1473140348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 0x7f4468102220 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4458004940 tx=0x7f4458009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.212+0000 7f446cf48700 1 -- 172.21.15.77:0/1473140348 shutdown_connections 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.212+0000 7f446cf48700 1 --2- 172.21.15.77:0/1473140348 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44681027f0 0x7f44680fbe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.212+0000 7f446cf48700 1 --2- 172.21.15.77:0/1473140348 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 0x7f4468102220 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.212+0000 7f446cf48700 1 -- 172.21.15.77:0/1473140348 >> 172.21.15.77:0/1473140348 conn(0x7f44680f79e0 msgr2=0x7f44680f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.212+0000 7f446cf48700 1 -- 172.21.15.77:0/1473140348 shutdown_connections 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.212+0000 7f446cf48700 1 -- 172.21.15.77:0/1473140348 wait complete. 2024-02-19T18:57:50.231 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446cf48700 1 Processor -- start 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446cf48700 1 -- start start 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446cf48700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 0x7f446810eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446cf48700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44681027f0 0x7f446810f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446cf48700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446810f5d0 con 0x7f4468101e50 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446cf48700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f446810f740 con 0x7f44681027f0 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 0x7f446810eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 0x7f446810eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54046/0 (socket says 172.21.15.77:54046) 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446659c700 1 -- 172.21.15.77:0/1001101442 learned_addr learned my addr 172.21.15.77:0/1001101442 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:50.232 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f4465d9b700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44681027f0 0x7f446810f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446659c700 1 -- 172.21.15.77:0/1001101442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44681027f0 msgr2=0x7f446810f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446659c700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44681027f0 0x7f446810f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446659c700 1 -- 172.21.15.77:0/1001101442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458005040 con 0x7f4468101e50 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446659c700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 0x7f446810eb00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f445800b3b0 tx=0x7f4458009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f4456ffd700 1 -- 172.21.15.77:0/1001101442 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445800e040 con 0x7f4468101e50 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4468117a70 con 0x7f4468101e50 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f4456ffd700 1 -- 172.21.15.77:0/1001101442 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f445800bbf0 con 0x7f4468101e50 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.213+0000 7f4456ffd700 1 -- 172.21.15.77:0/1001101442 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f445801c3f0 con 0x7f4468101e50 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.214+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4468117f30 con 0x7f4468101e50 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.215+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44680ff160 con 0x7f4468101e50 2024-02-19T18:57:50.234 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.215+0000 7f4456ffd700 1 -- 172.21.15.77:0/1001101442 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f445800bd60 con 0x7f4468101e50 2024-02-19T18:57:50.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.215+0000 7f4456ffd700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f44500692a0 0x7f445006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:50.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.216+0000 7f4456ffd700 1 -- 172.21.15.77:0/1001101442 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f44580881b0 con 0x7f4468101e50 2024-02-19T18:57:50.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.216+0000 7f4465d9b700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f44500692a0 0x7f445006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:50.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.216+0000 7f4465d9b700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f44500692a0 0x7f445006b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4468066a30 tx=0x7f445c00b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:50.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.220+0000 7f4456ffd700 1 -- 172.21.15.77:0/1001101442 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4458054820 con 0x7f4468101e50 2024-02-19T18:57:50.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:49 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3781750273' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-19T18:57:50.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:57:50.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:49 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3513693299' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-19T18:57:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.409+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4468064b00 con 0x7f4468101e50 2024-02-19T18:57:50.427 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.409+0000 7f4456ffd700 1 -- 172.21.15.77:0/1001101442 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4458057e40 con 0x7f4468101e50 2024-02-19T18:57:50.427 INFO:teuthology.orchestra.run.smithi077.stdout:60129542178 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f44500692a0 msgr2=0x7f445006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f44500692a0 0x7f445006b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4468066a30 tx=0x7f445c00b040 comp rx=0 tx=0).stop 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 msgr2=0x7f446810eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 0x7f446810eb00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f445800b3b0 tx=0x7f4458009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 shutdown_connections 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f44500692a0 0x7f445006b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f4468101e50 0x7f446810eb00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 --2- 172.21.15.77:0/1001101442 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f44681027f0 0x7f446810f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 >> 172.21.15.77:0/1001101442 conn(0x7f44680f79e0 msgr2=0x7f44680f9450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 shutdown_connections 2024-02-19T18:57:50.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:50.412+0000 7f446cf48700 1 -- 172.21.15.77:0/1001101442 wait complete. 2024-02-19T18:57:50.907 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378651 for osd.2 2024-02-19T18:57:50.907 DEBUG:teuthology.parallel:result is None 2024-02-19T18:57:51.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:50 smithi077 ceph-mon[29247]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:51.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:50 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1001101442' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T18:57:51.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:50 smithi104 ceph-mon[42320]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:51.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:50 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1001101442' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-19T18:57:51.407 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542178 for osd.1 2024-02-19T18:57:51.407 DEBUG:teuthology.parallel:result is None 2024-02-19T18:57:51.802 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691850 got 171798691852 for osd.5 2024-02-19T18:57:51.802 DEBUG:teuthology.parallel:result is None 2024-02-19T18:57:51.802 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-19T18:57:51.802 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T18:57:52.056 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:53.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:53 smithi077 ceph-mon[29247]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:53.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:53 smithi104 ceph-mon[42320]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:53.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.817+0000 7f2428647700 1 -- 172.21.15.77:0/3984563773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24201027f0 msgr2=0x7f24200fbe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:53.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.817+0000 7f2428647700 1 --2- 172.21.15.77:0/3984563773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24201027f0 0x7f24200fbe60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2410004940 tx=0x7f2410009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:53.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.817+0000 7f2428647700 1 -- 172.21.15.77:0/3984563773 shutdown_connections 2024-02-19T18:57:53.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.817+0000 7f2428647700 1 --2- 172.21.15.77:0/3984563773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f24201027f0 0x7f24200fbe60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:53.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.817+0000 7f2428647700 1 --2- 172.21.15.77:0/3984563773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2420101e50 0x7f2420102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:53.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.817+0000 7f2428647700 1 -- 172.21.15.77:0/3984563773 >> 172.21.15.77:0/3984563773 conn(0x7f24200f79e0 msgr2=0x7f24200f9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:53.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.817+0000 7f2428647700 1 -- 172.21.15.77:0/3984563773 shutdown_connections 2024-02-19T18:57:53.835 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.817+0000 7f2428647700 1 -- 172.21.15.77:0/3984563773 wait complete. 2024-02-19T18:57:53.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f2428647700 1 Processor -- start 2024-02-19T18:57:53.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f2428647700 1 -- start start 2024-02-19T18:57:53.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f2428647700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2420101e50 0x7f242010eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:53.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f2428647700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24201027f0 0x7f242010f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:53.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f2428647700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242010f600 con 0x7f2420101e50 2024-02-19T18:57:53.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f2428647700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242010f740 con 0x7f24201027f0 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f24263e3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2420101e50 0x7f242010eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f24263e3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2420101e50 0x7f242010eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:54074/0 (socket says 172.21.15.77:54074) 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f24263e3700 1 -- 172.21.15.77:0/2561225780 learned_addr learned my addr 172.21.15.77:0/2561225780 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f2425be2700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24201027f0 0x7f242010f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f24263e3700 1 -- 172.21.15.77:0/2561225780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24201027f0 msgr2=0x7f242010f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f24263e3700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24201027f0 0x7f242010f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f24263e3700 1 -- 172.21.15.77:0/2561225780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2410005040 con 0x7f2420101e50 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.818+0000 7f24263e3700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2420101e50 0x7f242010eb30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f241c002a50 tx=0x7f241c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.819+0000 7f2416ffd700 1 -- 172.21.15.77:0/2561225780 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c007cc0 con 0x7f2420101e50 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.819+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2420117aa0 con 0x7f2420101e50 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.819+0000 7f2416ffd700 1 -- 172.21.15.77:0/2561225780 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f241c00f040 con 0x7f2420101e50 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.819+0000 7f2416ffd700 1 -- 172.21.15.77:0/2561225780 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f241c00da30 con 0x7f2420101e50 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.819+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2420117fb0 con 0x7f2420101e50 2024-02-19T18:57:53.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.820+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2420104320 con 0x7f2420101e50 2024-02-19T18:57:53.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.820+0000 7f2416ffd700 1 -- 172.21.15.77:0/2561225780 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f241c00dbb0 con 0x7f2420101e50 2024-02-19T18:57:53.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.820+0000 7f2416ffd700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f240c0692a0 0x7f240c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:53.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.820+0000 7f2416ffd700 1 -- 172.21.15.77:0/2561225780 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f241c086bd0 con 0x7f2420101e50 2024-02-19T18:57:53.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.821+0000 7f2425be2700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f240c0692a0 0x7f240c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:53.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.821+0000 7f2425be2700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f240c0692a0 0x7f240c06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f241000b3b0 tx=0x7f2410009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:53.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:53.825+0000 7f2416ffd700 1 -- 172.21.15.77:0/2561225780 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f241c0531c0 con 0x7f2420101e50 2024-02-19T18:57:54.021 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.004+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f242004bf40 con 0x7f240c0692a0 2024-02-19T18:57:54.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.005+0000 7f2416ffd700 1 -- 172.21.15.77:0/2561225780 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28034 (secure 0 0 0) 0x7f242004bf40 con 0x7f240c0692a0 2024-02-19T18:57:54.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:54.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f240c0692a0 msgr2=0x7f240c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:54.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f240c0692a0 0x7f240c06b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f241000b3b0 tx=0x7f2410009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:54.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2420101e50 msgr2=0x7f242010eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2420101e50 0x7f242010eb30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f241c002a50 tx=0x7f241c0046b0 comp rx=0 tx=0).stop 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 shutdown_connections 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f240c0692a0 0x7f240c06b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2420101e50 0x7f242010eb30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 --2- 172.21.15.77:0/2561225780 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f24201027f0 0x7f242010f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 >> 172.21.15.77:0/2561225780 conn(0x7f24200f79e0 msgr2=0x7f24200f94d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 shutdown_connections 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:54.007+0000 7f2428647700 1 -- 172.21.15.77:0/2561225780 wait complete. 2024-02-19T18:57:54.026 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-02-19T18:57:54.615 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-02-19T18:57:52.136485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787648,"kb_used_data":3920,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683264,"statfs":{"total":575970213888,"available":574139662336,"internally_reserved":0,"allocated":4014080,"data_stored":2624143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002243"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":188,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-19T18:57:02.319721+0000","last_change":"2024-02-19T18:56:36.031181+0000","last_active":"2024-02-19T18:57:02.319721+0000","last_peered":"2024-02-19T18:57:02.319721+0000","last_clean":"2024-02-19T18:57:02.319721+0000","last_became_active":"2024-02-19T18:56:36.030837+0000","last_became_peered":"2024-02-19T18:56:36.030837+0000","last_unstale":"2024-02-19T18:57:02.319721+0000","last_undegraded":"2024-02-19T18:57:02.319721+0000","last_fullsized":"2024-02-19T18:57:02.319721+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T18:55:49.053419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T18:55:49.053419+0000","last_clean_scrub_stamp":"2024-02-19T18:55:49.053419+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T06:10:11.301440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]}]},{"osd":4,"up_from":35,"seq":150323855377,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon Feb 19 18:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.55600000000000005}]},{"osd":2,"last update":"Mon Feb 19 18:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.48199999999999998}]},{"osd":3,"last update":"Mon Feb 19 18:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.31},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.33300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298228,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446924,"statfs":{"total":95995035648,"available":95689650176,"internally_reserved":0,"allocated":962560,"data_stored":732272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":8.9710000000000001,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":8.9849999999999994,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Feb 19 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Feb 19 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52500000000000002}]},{"osd":4,"last update":"Mon Feb 19 18:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.59099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.38300000000000001}]},{"osd":1,"last update":"Mon Feb 19 18:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.314},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.41799999999999998}]},{"osd":3,"last update":"Mon Feb 19 18:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.72999999999999998}]},{"osd":4,"last update":"Mon Feb 19 18:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.31,"15min":0.30599999999999999},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.48299999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32300000000000001,"15min":0.317},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.252}]},{"osd":2,"last update":"Mon Feb 19 18:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.28899999999999998}]},{"osd":3,"last update":"Mon Feb 19 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.40899999999999997}]},{"osd":4,"last update":"Mon Feb 19 18:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.48699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:57:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33300000000000002,"15min":0.33000000000000002},"min":{"1min":0.25800000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.46500000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33400000000000002},"min":{"1min":0.23300000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.52500000000000002,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.45200000000000001}]},{"osd":2,"last update":"Mon Feb 19 18:56:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.46700000000000003}]},{"osd":3,"last update":"Mon Feb 19 18:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.69999999999999996}]},{"osd":4,"last update":"Mon Feb 19 18:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.626},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.71099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T18:57:54.616 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph pg dump --format=json 2024-02-19T18:57:54.869 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:55.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:55 smithi077 ceph-mon[29247]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:57:55.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:55 smithi077 ceph-mon[29247]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:55.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:55 smithi104 ceph-mon[42320]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:57:55.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:55 smithi104 ceph-mon[42320]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:56.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.462+0000 7fd62da2a700 1 -- 172.21.15.77:0/2036000470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fdf30 msgr2=0x7fd6280fe300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.462+0000 7fd62da2a700 1 --2- 172.21.15.77:0/2036000470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fdf30 0x7fd6280fe300 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd620004940 tx=0x7fd620009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.462+0000 7fd62da2a700 1 -- 172.21.15.77:0/2036000470 shutdown_connections 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.462+0000 7fd62da2a700 1 --2- 172.21.15.77:0/2036000470 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6280fe8d0 0x7fd628106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.462+0000 7fd62da2a700 1 --2- 172.21.15.77:0/2036000470 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fdf30 0x7fd6280fe300 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.462+0000 7fd62da2a700 1 -- 172.21.15.77:0/2036000470 >> 172.21.15.77:0/2036000470 conn(0x7fd6280f79d0 msgr2=0x7fd6280f9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.462+0000 7fd62da2a700 1 -- 172.21.15.77:0/2036000470 shutdown_connections 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.462+0000 7fd62da2a700 1 -- 172.21.15.77:0/2036000470 wait complete. 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd62da2a700 1 Processor -- start 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd62da2a700 1 -- start start 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd62da2a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6280fdf30 0x7fd628113ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:56.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd62da2a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fe8d0 0x7fd628114020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd62da2a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd628114700 con 0x7fd6280fe8d0 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd62da2a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62810d9a0 con 0x7fd6280fdf30 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd6267fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fe8d0 0x7fd628114020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd6267fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fe8d0 0x7fd628114020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46718/0 (socket says 172.21.15.77:46718) 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd6267fc700 1 -- 172.21.15.77:0/1839906265 learned_addr learned my addr 172.21.15.77:0/1839906265 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd626ffd700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6280fdf30 0x7fd628113ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd6267fc700 1 -- 172.21.15.77:0/1839906265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6280fdf30 msgr2=0x7fd628113ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd6267fc700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6280fdf30 0x7fd628113ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd6267fc700 1 -- 172.21.15.77:0/1839906265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd620005040 con 0x7fd6280fe8d0 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd6267fc700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fe8d0 0x7fd628114020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00ffa0 tx=0x7fd61c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.463+0000 7fd60f7fe700 1 -- 172.21.15.77:0/1839906265 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd61c0092e0 con 0x7fd6280fe8d0 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.464+0000 7fd60f7fe700 1 -- 172.21.15.77:0/1839906265 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd61c019470 con 0x7fd6280fe8d0 2024-02-19T18:57:56.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.464+0000 7fd60f7fe700 1 -- 172.21.15.77:0/1839906265 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd61c007640 con 0x7fd6280fe8d0 2024-02-19T18:57:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.464+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62810dc20 con 0x7fd6280fe8d0 2024-02-19T18:57:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.464+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62810e160 con 0x7fd6280fe8d0 2024-02-19T18:57:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.465+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd628104600 con 0x7fd6280fe8d0 2024-02-19T18:57:56.484 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.465+0000 7fd60f7fe700 1 -- 172.21.15.77:0/1839906265 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fd61c016070 con 0x7fd6280fe8d0 2024-02-19T18:57:56.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.465+0000 7fd60f7fe700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd6100692a0 0x7fd61006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:56.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.466+0000 7fd626ffd700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd6100692a0 0x7fd61006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:56.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.466+0000 7fd60f7fe700 1 -- 172.21.15.77:0/1839906265 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd61c08adf0 con 0x7fd6280fe8d0 2024-02-19T18:57:56.485 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.466+0000 7fd626ffd700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd6100692a0 0x7fd61006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd620004f90 tx=0x7fd620009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:56.487 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.470+0000 7fd60f7fe700 1 -- 172.21.15.77:0/1839906265 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fd61c05aef0 con 0x7fd6280fe8d0 2024-02-19T18:57:56.670 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.652+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd62810a040 con 0x7fd6100692a0 2024-02-19T18:57:56.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.654+0000 7fd60f7fe700 1 -- 172.21.15.77:0/1839906265 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28062 (secure 0 0 0) 0x7fd62810a040 con 0x7fd6100692a0 2024-02-19T18:57:56.672 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd6100692a0 msgr2=0x7fd61006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd6100692a0 0x7fd61006b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd620004f90 tx=0x7fd620009e30 comp rx=0 tx=0).stop 2024-02-19T18:57:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fe8d0 msgr2=0x7fd628114020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fe8d0 0x7fd628114020 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00ffa0 tx=0x7fd61c009840 comp rx=0 tx=0).stop 2024-02-19T18:57:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 shutdown_connections 2024-02-19T18:57:56.674 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fd6100692a0 0x7fd61006b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd6280fdf30 0x7fd628113ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 --2- 172.21.15.77:0/1839906265 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd6280fe8d0 0x7fd628114020 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 >> 172.21.15.77:0/1839906265 conn(0x7fd6280f79d0 msgr2=0x7fd628101780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 shutdown_connections 2024-02-19T18:57:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:56.656+0000 7fd62da2a700 1 -- 172.21.15.77:0/1839906265 wait complete. 2024-02-19T18:57:56.675 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-02-19T18:57:57.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:57 smithi077 ceph-mon[29247]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:57.238 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:57 smithi077 ceph-mon[29247]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:57:57.262 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":160,"stamp":"2024-02-19T18:57:56.137139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787648,"kb_used_data":3920,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683264,"statfs":{"total":575970213888,"available":574139662336,"internally_reserved":0,"allocated":4014080,"data_stored":2624143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002230"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":188,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-02-19T18:57:02.319721+0000","last_change":"2024-02-19T18:56:36.031181+0000","last_active":"2024-02-19T18:57:02.319721+0000","last_peered":"2024-02-19T18:57:02.319721+0000","last_clean":"2024-02-19T18:57:02.319721+0000","last_became_active":"2024-02-19T18:56:36.030837+0000","last_became_peered":"2024-02-19T18:56:36.030837+0000","last_unstale":"2024-02-19T18:57:02.319721+0000","last_undegraded":"2024-02-19T18:57:02.319721+0000","last_fullsized":"2024-02-19T18:57:02.319721+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-19T18:55:49.053419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-19T18:55:49.053419+0000","last_clean_scrub_stamp":"2024-02-19T18:55:49.053419+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-21T06:10:11.301440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.58199999999999996}]},{"osd":1,"last update":"Mon Feb 19 18:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.59899999999999998}]},{"osd":2,"last update":"Mon Feb 19 18:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Feb 19 18:57:40 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298228,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446924,"statfs":{"total":95995035648,"available":95689650176,"internally_reserved":0,"allocated":962560,"data_stored":732272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":8.9710000000000001,"5min":8.9710000000000001,"15min":8.9710000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":8.9849999999999994,"5min":8.9849999999999994,"15min":8.9849999999999994},"last":0.40799999999999997}]},{"osd":1,"last update":"Mon Feb 19 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.54700000000000004}]},{"osd":2,"last update":"Mon Feb 19 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52500000000000002}]},{"osd":4,"last update":"Mon Feb 19 18:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.59099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"last":0.25},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.26500000000000001}]},{"osd":1,"last update":"Mon Feb 19 18:56:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.28699999999999998}]},{"osd":3,"last update":"Mon Feb 19 18:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.63100000000000001}]},{"osd":4,"last update":"Mon Feb 19 18:57:38 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.61599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705705,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Feb 19 18:57:21 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.31,"15min":0.30599999999999999},"min":{"1min":0.23200000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.48299999999999998,"5min":0.51400000000000001,"15min":0.51400000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.32300000000000001,"15min":0.317},"min":{"1min":0.22500000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.29699999999999999}]},{"osd":2,"last update":"Mon Feb 19 18:56:51 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.316},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.33000000000000002}]},{"osd":3,"last update":"Mon Feb 19 18:57:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Feb 19 18:57:37 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.59599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Feb 19 18:57:17 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.33300000000000002,"15min":0.33000000000000002},"min":{"1min":0.25800000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.46500000000000002,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.33500000000000002,"15min":0.33400000000000002},"min":{"1min":0.23300000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.52500000000000002,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.26000000000000001}]},{"osd":2,"last update":"Mon Feb 19 18:56:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.27200000000000002}]},{"osd":3,"last update":"Mon Feb 19 18:57:12 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Feb 19 18:57:39 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-19T18:57:57.263 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-19T18:57:57.263 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-19T18:57:57.263 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-19T18:57:57.263 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph health --format=json 2024-02-19T18:57:57.515 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:57:57.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:57 smithi104 ceph-mon[42320]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:57.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:57 smithi104 ceph-mon[42320]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-19T18:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 -- 172.21.15.77:0/3601422397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c103140 msgr2=0x7f784c103510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 --2- 172.21.15.77:0/3601422397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c103140 0x7f784c103510 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7844009ce0 tx=0x7f7844004d10 comp rx=0 tx=0).stop 2024-02-19T18:57:58.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 -- 172.21.15.77:0/3601422397 shutdown_connections 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 --2- 172.21.15.77:0/3601422397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f784c0fb570 0x7f784c0fb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 --2- 172.21.15.77:0/3601422397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c103140 0x7f784c103510 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 -- 172.21.15.77:0/3601422397 >> 172.21.15.77:0/3601422397 conn(0x7f784c071560 msgr2=0x7f784c071960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 -- 172.21.15.77:0/3601422397 shutdown_connections 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 -- 172.21.15.77:0/3601422397 wait complete. 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 Processor -- start 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.892+0000 7f78534a8700 1 -- start start 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f78534a8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c0fb570 0x7f784c113cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f78534a8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f784c103140 0x7f784c1141f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f78534a8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c114840 con 0x7f784c0fb570 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f78534a8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c10db50 con 0x7f784c103140 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7851244700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c0fb570 0x7f784c113cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7851244700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c0fb570 0x7f784c113cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46732/0 (socket says 172.21.15.77:46732) 2024-02-19T18:57:58.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7851244700 1 -- 172.21.15.77:0/765270009 learned_addr learned my addr 172.21.15.77:0/765270009 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7850a43700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f784c103140 0x7f784c1141f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7851244700 1 -- 172.21.15.77:0/765270009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f784c103140 msgr2=0x7f784c1141f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7851244700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f784c103140 0x7f784c1141f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7851244700 1 -- 172.21.15.77:0/765270009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7844009940 con 0x7f784c0fb570 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7851244700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c0fb570 0x7f784c113cb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f784400bf60 tx=0x7f7844004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7841ffb700 1 -- 172.21.15.77:0/765270009 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f784400e040 con 0x7f784c0fb570 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784c10ddd0 con 0x7f784c0fb570 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7841ffb700 1 -- 172.21.15.77:0/765270009 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78440128e0 con 0x7f784c0fb570 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.893+0000 7f7841ffb700 1 -- 172.21.15.77:0/765270009 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f784401c870 con 0x7f784c0fb570 2024-02-19T18:57:58.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.894+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784c10e310 con 0x7f784c0fb570 2024-02-19T18:57:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.895+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784c100310 con 0x7f784c0fb570 2024-02-19T18:57:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.895+0000 7f7841ffb700 1 -- 172.21.15.77:0/765270009 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7844012a50 con 0x7f784c0fb570 2024-02-19T18:57:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.895+0000 7f7841ffb700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f78380692a0 0x7f783806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:57:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.896+0000 7f7841ffb700 1 -- 172.21.15.77:0/765270009 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7844088bc0 con 0x7f784c0fb570 2024-02-19T18:57:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.896+0000 7f7850a43700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f78380692a0 0x7f783806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:57:58.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.896+0000 7f7850a43700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f78380692a0 0x7f783806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f783c007fd0 tx=0x7f783c009040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:57:58.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:58.900+0000 7f7841ffb700 1 -- 172.21.15.77:0/765270009 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7844058c40 con 0x7f784c0fb570 2024-02-19T18:57:59.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:58 smithi077 ceph-mon[29247]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.125+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f784c073080 con 0x7f784c0fb570 2024-02-19T18:57:59.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.126+0000 7f7841ffb700 1 -- 172.21.15.77:0/765270009 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f78440587d0 con 0x7f784c0fb570 2024-02-19T18:57:59.144 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:57:59.144 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f78380692a0 msgr2=0x7f783806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f78380692a0 0x7f783806b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f783c007fd0 tx=0x7f783c009040 comp rx=0 tx=0).stop 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c0fb570 msgr2=0x7f784c113cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c0fb570 0x7f784c113cb0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f784400bf60 tx=0x7f7844004d10 comp rx=0 tx=0).stop 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 shutdown_connections 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f78380692a0 0x7f783806b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f784c0fb570 0x7f784c113cb0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 --2- 172.21.15.77:0/765270009 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f784c103140 0x7f784c1141f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 >> 172.21.15.77:0/765270009 conn(0x7f784c071560 msgr2=0x7f784c0fa680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 shutdown_connections 2024-02-19T18:57:59.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:57:59.128+0000 7f78534a8700 1 -- 172.21.15.77:0/765270009 wait complete. 2024-02-19T18:57:59.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:58 smithi104 ceph-mon[42320]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:57:59.767 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-19T18:57:59.768 INFO:tasks.cephadm:Setup complete, yielding 2024-02-19T18:57:59.768 INFO:teuthology.run_tasks:Running task print... 2024-02-19T18:57:59.777 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-02-19T18:57:59.778 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:57:59.789 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:57:59.789 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-02-19T18:58:00.043 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:00.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:57:59 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/765270009' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T18:58:00.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:57:59 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/765270009' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-19T18:58:01.174 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:00 smithi077 ceph-mon[29247]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:01.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:00 smithi104 ceph-mon[42320]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:01.630 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 -- 172.21.15.77:0/1578253855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 msgr2=0x7fc95c0ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 --2- 172.21.15.77:0/1578253855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 0x7fc95c0ffe60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc94c004940 tx=0x7fc94c009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 -- 172.21.15.77:0/1578253855 shutdown_connections 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 --2- 172.21.15.77:0/1578253855 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc95c1003a0 0x7fc95c102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 --2- 172.21.15.77:0/1578253855 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 0x7fc95c0ffe60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 -- 172.21.15.77:0/1578253855 >> 172.21.15.77:0/1578253855 conn(0x7fc95c0f74e0 msgr2=0x7fc95c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 -- 172.21.15.77:0/1578253855 shutdown_connections 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 -- 172.21.15.77:0/1578253855 wait complete. 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.611+0000 7fc961295700 1 Processor -- start 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc961295700 1 -- start start 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc961295700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 0x7fc95c10d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:01.631 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc961295700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc95c1003a0 0x7fc95c10de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc961295700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c10e4e0 con 0x7fc95c0fda80 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc961295700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95c10ed20 con 0x7fc95c1003a0 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc95affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 0x7fc95c10d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc95affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 0x7fc95c10d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46748/0 (socket says 172.21.15.77:46748) 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc95affd700 1 -- 172.21.15.77:0/2890539380 learned_addr learned my addr 172.21.15.77:0/2890539380 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc9527fc700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc95c1003a0 0x7fc95c10de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc95affd700 1 -- 172.21.15.77:0/2890539380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc95c1003a0 msgr2=0x7fc95c10de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc95affd700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc95c1003a0 0x7fc95c10de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc95affd700 1 -- 172.21.15.77:0/2890539380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc94c005040 con 0x7fc95c0fda80 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc95affd700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 0x7fc95c10d910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0093b0 tx=0x7fc94c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc953fff700 1 -- 172.21.15.77:0/2890539380 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc94c00e040 con 0x7fc95c0fda80 2024-02-19T18:58:01.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc953fff700 1 -- 172.21.15.77:0/2890539380 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc94c00bcc0 con 0x7fc95c0fda80 2024-02-19T18:58:01.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc95c10efa0 con 0x7fc95c0fda80 2024-02-19T18:58:01.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.612+0000 7fc953fff700 1 -- 172.21.15.77:0/2890539380 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc94c01c630 con 0x7fc95c0fda80 2024-02-19T18:58:01.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.613+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc95c10f4e0 con 0x7fc95c0fda80 2024-02-19T18:58:01.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.613+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc95c107ad0 con 0x7fc95c0fda80 2024-02-19T18:58:01.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.614+0000 7fc953fff700 1 -- 172.21.15.77:0/2890539380 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fc94c0139f0 con 0x7fc95c0fda80 2024-02-19T18:58:01.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.614+0000 7fc953fff700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc9480692a0 0x7fc94806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:01.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.614+0000 7fc953fff700 1 -- 172.21.15.77:0/2890539380 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc94c088250 con 0x7fc95c0fda80 2024-02-19T18:58:01.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.614+0000 7fc9527fc700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc9480692a0 0x7fc94806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:01.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.615+0000 7fc9527fc700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc9480692a0 0x7fc94806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc944006fd0 tx=0x7fc944008040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:01.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.619+0000 7fc953fff700 1 -- 172.21.15.77:0/2890539380 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fc94c058350 con 0x7fc95c0fda80 2024-02-19T18:58:01.820 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.802+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7fc95c04bed0 con 0x7fc95c0fda80 2024-02-19T18:58:01.824 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.807+0000 7fc953fff700 1 -- 172.21.15.77:0/2890539380 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7fc94c057ee0 con 0x7fc95c0fda80 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.809+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc9480692a0 msgr2=0x7fc94806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.809+0000 7fc961295700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc9480692a0 0x7fc94806b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc944006fd0 tx=0x7fc944008040 comp rx=0 tx=0).stop 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.809+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 msgr2=0x7fc95c10d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.809+0000 7fc961295700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 0x7fc95c10d910 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc94c0093b0 tx=0x7fc94c009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.810+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 shutdown_connections 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.810+0000 7fc961295700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fc9480692a0 0x7fc94806b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.810+0000 7fc961295700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fc95c0fda80 0x7fc95c10d910 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.810+0000 7fc961295700 1 --2- 172.21.15.77:0/2890539380 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fc95c1003a0 0x7fc95c10de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.810+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 >> 172.21.15.77:0/2890539380 conn(0x7fc95c0f74e0 msgr2=0x7fc95c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.810+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 shutdown_connections 2024-02-19T18:58:01.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:01.810+0000 7fc961295700 1 -- 172.21.15.77:0/2890539380 wait complete. 2024-02-19T18:58:02.579 INFO:teuthology.run_tasks:Running task print... 2024-02-19T18:58:02.588 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-02-19T18:58:02.588 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:58:02.613 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:58:02.613 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch status' 2024-02-19T18:58:02.882 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:03.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:02 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/2890539380' entity='client.admin' 2024-02-19T18:58:03.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:02 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:03.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:02 smithi077 ceph-mon[29247]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:03.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:02 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:03.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:02 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:03.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:02 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:03.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:02 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/2890539380' entity='client.admin' 2024-02-19T18:58:03.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:02 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:03.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:02 smithi104 ceph-mon[42320]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:03.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:02 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:03.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:02 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:03.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:02 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.412+0000 7f3ecdb77700 1 -- 172.21.15.77:0/1817751058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fcef0 msgr2=0x7f3ec80fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.412+0000 7f3ecdb77700 1 --2- 172.21.15.77:0/1817751058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fcef0 0x7f3ec80fd320 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3eb8004940 tx=0x7f3eb8009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.412+0000 7f3ecdb77700 1 -- 172.21.15.77:0/1817751058 shutdown_connections 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.412+0000 7f3ecdb77700 1 --2- 172.21.15.77:0/1817751058 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec80fe0e0 0x7f3ec80fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.412+0000 7f3ecdb77700 1 --2- 172.21.15.77:0/1817751058 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fcef0 0x7f3ec80fd320 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.412+0000 7f3ecdb77700 1 -- 172.21.15.77:0/1817751058 >> 172.21.15.77:0/1817751058 conn(0x7f3ec80f8620 msgr2=0x7f3ec80faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.412+0000 7f3ecdb77700 1 -- 172.21.15.77:0/1817751058 shutdown_connections 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.412+0000 7f3ecdb77700 1 -- 172.21.15.77:0/1817751058 wait complete. 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ecdb77700 1 Processor -- start 2024-02-19T18:58:04.432 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ecdb77700 1 -- start start 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ecdb77700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec80fcef0 0x7f3ec810d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ecdb77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fe0e0 0x7f3ec810def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ecdb77700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec810e580 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ecdb77700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec810edc0 con 0x7f3ec80fcef0 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ebeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fe0e0 0x7f3ec810def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ebeffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fe0e0 0x7f3ec810def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46770/0 (socket says 172.21.15.77:46770) 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ebeffd700 1 -- 172.21.15.77:0/3256385586 learned_addr learned my addr 172.21.15.77:0/3256385586 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ec77fe700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec80fcef0 0x7f3ec810d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ebeffd700 1 -- 172.21.15.77:0/3256385586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec80fcef0 msgr2=0x7f3ec810d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ebeffd700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec80fcef0 0x7f3ec810d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ebeffd700 1 -- 172.21.15.77:0/3256385586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb8005040 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.413+0000 7f3ec77fe700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec80fcef0 0x7f3ec810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.414+0000 7f3ebeffd700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fe0e0 0x7f3ec810def0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000b790 tx=0x7f3eb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.414+0000 7f3ec4ff9700 1 -- 172.21.15.77:0/3256385586 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3eb000f770 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.414+0000 7f3ec4ff9700 1 -- 172.21.15.77:0/3256385586 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3eb000fda0 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.414+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec810f0a0 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.414+0000 7f3ec4ff9700 1 -- 172.21.15.77:0/3256385586 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3eb000d310 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.414+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec810f5e0 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.434 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.415+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec804bed0 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.415+0000 7f3ec4ff9700 1 -- 172.21.15.77:0/3256385586 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3eb0010420 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.415+0000 7f3ec4ff9700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3eb40692b0 0x7f3eb406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:04.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.416+0000 7f3ec4ff9700 1 -- 172.21.15.77:0/3256385586 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3eb0086e70 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.416+0000 7f3ec77fe700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3eb40692b0 0x7f3eb406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:04.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.416+0000 7f3ec77fe700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3eb40692b0 0x7f3eb406b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800b3b0 tx=0x7f3eb8009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:04.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.420+0000 7f3ec4ff9700 1 -- 172.21.15.77:0/3256385586 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f3eb0056f70 con 0x7f3ec80fe0e0 2024-02-19T18:58:04.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:04.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:04.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.617+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ec810fab0 con 0x7f3eb40692b0 2024-02-19T18:58:04.636 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.619+0000 7f3ec4ff9700 1 -- 172.21.15.77:0/3256385586 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f3ec810fab0 con 0x7f3eb40692b0 2024-02-19T18:58:04.636 INFO:teuthology.orchestra.run.smithi077.stdout:Backend: cephadm 2024-02-19T18:58:04.637 INFO:teuthology.orchestra.run.smithi077.stdout:Available: Yes 2024-02-19T18:58:04.637 INFO:teuthology.orchestra.run.smithi077.stdout:Paused: No 2024-02-19T18:58:04.638 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.620+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3eb40692b0 msgr2=0x7f3eb406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.620+0000 7f3ecdb77700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3eb40692b0 0x7f3eb406b760 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800b3b0 tx=0x7f3eb8009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.620+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fe0e0 msgr2=0x7f3ec810def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.620+0000 7f3ecdb77700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fe0e0 0x7f3ec810def0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3eb000b790 tx=0x7f3eb0009250 comp rx=0 tx=0).stop 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.620+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 shutdown_connections 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.620+0000 7f3ecdb77700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f3eb40692b0 0x7f3eb406b760 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.620+0000 7f3ecdb77700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f3ec80fcef0 0x7f3ec810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.620+0000 7f3ecdb77700 1 --2- 172.21.15.77:0/3256385586 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f3ec80fe0e0 0x7f3ec810def0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.621+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 >> 172.21.15.77:0/3256385586 conn(0x7f3ec80f8620 msgr2=0x7f3ec81013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.621+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 shutdown_connections 2024-02-19T18:58:04.639 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:04.621+0000 7f3ecdb77700 1 -- 172.21.15.77:0/3256385586 wait complete. 2024-02-19T18:58:05.194 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-19T18:58:05.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:05 smithi077 ceph-mon[29247]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:05.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:05 smithi077 ceph-mon[29247]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:05.457 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:05.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:05 smithi104 ceph-mon[42320]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:05.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:05 smithi104 ceph-mon[42320]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:07.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.082+0000 7fb04c723700 1 -- 172.21.15.77:0/808228887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 msgr2=0x7fb0440fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:07.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.082+0000 7fb04c723700 1 --2- 172.21.15.77:0/808228887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 0x7fb0440fc250 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb034004940 tx=0x7fb034009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:07.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.082+0000 7fb04c723700 1 -- 172.21.15.77:0/808228887 shutdown_connections 2024-02-19T18:58:07.102 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.082+0000 7fb04c723700 1 --2- 172.21.15.77:0/808228887 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0440fcfb0 0x7fb0440fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.082+0000 7fb04c723700 1 --2- 172.21.15.77:0/808228887 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 0x7fb0440fc250 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.082+0000 7fb04c723700 1 -- 172.21.15.77:0/808228887 >> 172.21.15.77:0/808228887 conn(0x7fb0440f7520 msgr2=0x7fb0440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.082+0000 7fb04c723700 1 -- 172.21.15.77:0/808228887 shutdown_connections 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.082+0000 7fb04c723700 1 -- 172.21.15.77:0/808228887 wait complete. 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04c723700 1 Processor -- start 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04c723700 1 -- start start 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04c723700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 0x7fb0440fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04c723700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0440fcfb0 0x7fb0440fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04c723700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440fe980 con 0x7fb0440fbe40 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04c723700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440feaf0 con 0x7fb0440fcfb0 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04a4bf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 0x7fb0440fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04a4bf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 0x7fb0440fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58516/0 (socket says 172.21.15.77:58516) 2024-02-19T18:58:07.103 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04a4bf700 1 -- 172.21.15.77:0/27734504 learned_addr learned my addr 172.21.15.77:0/27734504 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb049cbe700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0440fcfb0 0x7fb0440fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04a4bf700 1 -- 172.21.15.77:0/27734504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0440fcfb0 msgr2=0x7fb0440fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04a4bf700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0440fcfb0 0x7fb0440fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04a4bf700 1 -- 172.21.15.77:0/27734504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034005040 con 0x7fb0440fbe40 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb04a4bf700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 0x7fb0440fff00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb0340093b0 tx=0x7fb034009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.083+0000 7fb03affd700 1 -- 172.21.15.77:0/27734504 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb03400e040 con 0x7fb0440fbe40 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.084+0000 7fb03affd700 1 -- 172.21.15.77:0/27734504 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb03400bcc0 con 0x7fb0440fbe40 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.084+0000 7fb03affd700 1 -- 172.21.15.77:0/27734504 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb03401c630 con 0x7fb0440fbe40 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.084+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0440fed70 con 0x7fb0440fbe40 2024-02-19T18:58:07.104 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.084+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb044115960 con 0x7fb0440fbe40 2024-02-19T18:58:07.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.085+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb04404bed0 con 0x7fb0440fbe40 2024-02-19T18:58:07.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.085+0000 7fb03affd700 1 -- 172.21.15.77:0/27734504 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb0340139f0 con 0x7fb0440fbe40 2024-02-19T18:58:07.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.085+0000 7fb03affd700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb0300692a0 0x7fb03006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:07.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.086+0000 7fb03affd700 1 -- 172.21.15.77:0/27734504 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb0340882a0 con 0x7fb0440fbe40 2024-02-19T18:58:07.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.086+0000 7fb049cbe700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb0300692a0 0x7fb03006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:07.105 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.086+0000 7fb049cbe700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb0300692a0 0x7fb03006b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb040007fd0 tx=0x7fb040009040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:07.107 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.090+0000 7fb03affd700 1 -- 172.21.15.77:0/27734504 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb0340583a0 con 0x7fb0440fbe40 2024-02-19T18:58:07.299 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.281+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fb044066660 con 0x7fb0300692a0 2024-02-19T18:58:07.307 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.289+0000 7fb03affd700 1 -- 172.21.15.77:0/27734504 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7fb044066660 con 0x7fb0300692a0 2024-02-19T18:58:07.307 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T18:58:07.307 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (3m) 2m ago 6m 27.7M - 0.25.0 c8568f914cd2 1e676c9ed794 2024-02-19T18:58:07.307 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (7m) 2m ago 7m 6656k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T18:58:07.307 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (4m) 67s ago 4m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T18:58:07.307 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (3m) 2m ago 5m 81.6M - 9.4.7 954c08fa6188 2cb428963663 2024-02-19T18:58:07.307 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:9283 running (8m) 2m ago 8m 471M - 17.2.7-521-gc80e7168 10b0c032c913 78483a70884c 2024-02-19T18:58:07.307 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283 running (4m) 67s ago 4m 434M - 17.2.7-521-gc80e7168 10b0c032c913 5693aa48a572 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (8m) 2m ago 8m 45.6M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (4m) 67s ago 4m 34.5M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (6m) 2m ago 6m 14.2M - 1.5.0 0da6a335fe13 18fa31ab4762 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (4m) 67s ago 4m 15.1M - 1.5.0 0da6a335fe13 5ec125749863 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (3m) 2m ago 3m 37.3M 4288M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (2m) 2m ago 2m 37.4M 4288M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (2m) 2m ago 2m 37.2M 4288M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (2m) 67s ago 2m 39.7M 5312M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (98s) 67s ago 98s 39.1M 5312M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (72s) 67s ago 71s 11.0M 5312M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T18:58:07.308 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (3m) 2m ago 4m 38.0M - 2.43.0 a07b618ecd1d 6dc30fbe4b32 2024-02-19T18:58:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb0300692a0 msgr2=0x7fb03006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb0300692a0 0x7fb03006b750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb040007fd0 tx=0x7fb040009040 comp rx=0 tx=0).stop 2024-02-19T18:58:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 msgr2=0x7fb0440fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 0x7fb0440fff00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb0340093b0 tx=0x7fb034009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 shutdown_connections 2024-02-19T18:58:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb0300692a0 0x7fb03006b750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb0440fbe40 0x7fb0440fff00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:07.309 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 --2- 172.21.15.77:0/27734504 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb0440fcfb0 0x7fb0440fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 >> 172.21.15.77:0/27734504 conn(0x7fb0440f7520 msgr2=0x7fb044065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 shutdown_connections 2024-02-19T18:58:07.310 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:07.291+0000 7fb04c723700 1 -- 172.21.15.77:0/27734504 wait complete. 2024-02-19T18:58:07.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:07 smithi077 ceph-mon[29247]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:07.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:07 smithi104 ceph-mon[42320]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:08.065 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-19T18:58:08.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:08 smithi077 ceph-mon[29247]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:08.318 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:08.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:08 smithi104 ceph-mon[42320]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:09.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:09 smithi077 ceph-mon[29247]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:09.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:09 smithi104 ceph-mon[42320]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:09.800 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.779+0000 7f136c1e5700 1 -- 172.21.15.77:0/746885292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fe0e0 msgr2=0x7f13640fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.779+0000 7f136c1e5700 1 --2- 172.21.15.77:0/746885292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fe0e0 0x7f13640fe550 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1360004940 tx=0x7f1360009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.779+0000 7f136c1e5700 1 -- 172.21.15.77:0/746885292 shutdown_connections 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.779+0000 7f136c1e5700 1 --2- 172.21.15.77:0/746885292 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fe0e0 0x7f13640fe550 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.779+0000 7f136c1e5700 1 --2- 172.21.15.77:0/746885292 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13640fcef0 0x7f13640fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.779+0000 7f136c1e5700 1 -- 172.21.15.77:0/746885292 >> 172.21.15.77:0/746885292 conn(0x7f13640f8620 msgr2=0x7f13640faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.779+0000 7f136c1e5700 1 -- 172.21.15.77:0/746885292 shutdown_connections 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.779+0000 7f136c1e5700 1 -- 172.21.15.77:0/746885292 wait complete. 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f136c1e5700 1 Processor -- start 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f136c1e5700 1 -- start start 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f136c1e5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fcef0 0x7f1364071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:09.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f136c1e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13640fe0e0 0x7f13640702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f136c1e5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1364070970 con 0x7f13640fcef0 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f136c1e5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1364070ae0 con 0x7f13640fe0e0 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369f81700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fcef0 0x7f1364071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369f81700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fcef0 0x7f1364071e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58536/0 (socket says 172.21.15.77:58536) 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369f81700 1 -- 172.21.15.77:0/2909730745 learned_addr learned my addr 172.21.15.77:0/2909730745 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369780700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13640fe0e0 0x7f13640702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369f81700 1 -- 172.21.15.77:0/2909730745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13640fe0e0 msgr2=0x7f13640702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369f81700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13640fe0e0 0x7f13640702e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369f81700 1 -- 172.21.15.77:0/2909730745 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1360005040 con 0x7f13640fcef0 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369780700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13640fe0e0 0x7f13640702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.780+0000 7f1369f81700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fcef0 0x7f1364071e60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f135400b790 tx=0x7f1354009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.781+0000 7f135a7fc700 1 -- 172.21.15.77:0/2909730745 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f135400f6f0 con 0x7f13640fcef0 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.781+0000 7f135a7fc700 1 -- 172.21.15.77:0/2909730745 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f135400fd20 con 0x7f13640fcef0 2024-02-19T18:58:09.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.781+0000 7f135a7fc700 1 -- 172.21.15.77:0/2909730745 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13540103a0 con 0x7f13640fcef0 2024-02-19T18:58:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.781+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1364070dc0 con 0x7f13640fcef0 2024-02-19T18:58:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.781+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13641159c0 con 0x7f13640fcef0 2024-02-19T18:58:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.782+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136404bed0 con 0x7f13640fcef0 2024-02-19T18:58:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.782+0000 7f135a7fc700 1 -- 172.21.15.77:0/2909730745 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1354011400 con 0x7f13640fcef0 2024-02-19T18:58:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.782+0000 7f135a7fc700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f13500692a0 0x7f135006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.783+0000 7f135a7fc700 1 -- 172.21.15.77:0/2909730745 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1354086c90 con 0x7f13640fcef0 2024-02-19T18:58:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.783+0000 7f1369780700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f13500692a0 0x7f135006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:09.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.783+0000 7f1369780700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f13500692a0 0x7f135006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1360000c00 tx=0x7f1360009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:09.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.787+0000 7f135a7fc700 1 -- 172.21.15.77:0/2909730745 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1354056d10 con 0x7f13640fcef0 2024-02-19T18:58:09.995 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.977+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1364102ac0 con 0x7f13500692a0 2024-02-19T18:58:09.999 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.981+0000 7f135a7fc700 1 -- 172.21.15.77:0/2909730745 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1170 (secure 0 0 0) 0x7f1364102ac0 con 0x7f13500692a0 2024-02-19T18:58:09.999 INFO:teuthology.orchestra.run.smithi077.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-19T18:58:09.999 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager ?:9093,9094 1/1 2m ago 7m count:1 2024-02-19T18:58:09.999 INFO:teuthology.orchestra.run.smithi077.stdout:crash 2/2 2m ago 7m * 2024-02-19T18:58:09.999 INFO:teuthology.orchestra.run.smithi077.stdout:grafana ?:3000 1/1 2m ago 7m count:1 2024-02-19T18:58:10.000 INFO:teuthology.orchestra.run.smithi077.stdout:mgr 2/2 2m ago 7m count:2 2024-02-19T18:58:10.000 INFO:teuthology.orchestra.run.smithi077.stdout:mon 2/2 2m ago 6m smithi077:172.21.15.77=smithi077;smithi104:172.21.15.104=smithi104;count:2 2024-02-19T18:58:10.000 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter ?:9100 2/2 2m ago 7m * 2024-02-19T18:58:10.000 INFO:teuthology.orchestra.run.smithi077.stdout:osd 6 2m ago - 2024-02-19T18:58:10.000 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus ?:9095 1/1 2m ago 7m count:1 2024-02-19T18:58:10.001 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f13500692a0 msgr2=0x7f135006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f13500692a0 0x7f135006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1360000c00 tx=0x7f1360009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fcef0 msgr2=0x7f1364071e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fcef0 0x7f1364071e60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f135400b790 tx=0x7f1354009220 comp rx=0 tx=0).stop 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 shutdown_connections 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f13500692a0 0x7f135006b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f13640fcef0 0x7f1364071e60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 --2- 172.21.15.77:0/2909730745 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f13640fe0e0 0x7f13640702e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.983+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 >> 172.21.15.77:0/2909730745 conn(0x7f13640f8620 msgr2=0x7f13641013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.984+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 shutdown_connections 2024-02-19T18:58:10.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:09.984+0000 7f136c1e5700 1 -- 172.21.15.77:0/2909730745 wait complete. 2024-02-19T18:58:10.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-19T18:58:10.945 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:11.390 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:11 smithi077 ceph-mon[29247]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:11.391 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:11 smithi077 ceph-mon[29247]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:11.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:11 smithi104 ceph-mon[42320]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:11.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:11 smithi104 ceph-mon[42320]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:12.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.476+0000 7f8fd4b59700 1 -- 172.21.15.77:0/1449859316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 msgr2=0x7f8fd00ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:12.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.476+0000 7f8fd4b59700 1 --2- 172.21.15.77:0/1449859316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 0x7f8fd00ffe60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0004940 tx=0x7f8fc0009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.476+0000 7f8fd4b59700 1 -- 172.21.15.77:0/1449859316 shutdown_connections 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.476+0000 7f8fd4b59700 1 --2- 172.21.15.77:0/1449859316 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd01003a0 0x7f8fd0102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.476+0000 7f8fd4b59700 1 --2- 172.21.15.77:0/1449859316 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 0x7f8fd00ffe60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.476+0000 7f8fd4b59700 1 -- 172.21.15.77:0/1449859316 >> 172.21.15.77:0/1449859316 conn(0x7f8fd00f7520 msgr2=0x7f8fd00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.476+0000 7f8fd4b59700 1 -- 172.21.15.77:0/1449859316 shutdown_connections 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.476+0000 7f8fd4b59700 1 -- 172.21.15.77:0/1449859316 wait complete. 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fd4b59700 1 Processor -- start 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fd4b59700 1 -- start start 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fd4b59700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 0x7f8fd00fcf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fd4b59700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd01003a0 0x7f8fd00fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fd4b59700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00fba80 con 0x7f8fd00fda80 2024-02-19T18:58:12.497 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fd4b59700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00fbbf0 con 0x7f8fd01003a0 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fce59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 0x7f8fd00fcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fce59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 0x7f8fd00fcf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58552/0 (socket says 172.21.15.77:58552) 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fce59c700 1 -- 172.21.15.77:0/2822680940 learned_addr learned my addr 172.21.15.77:0/2822680940 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fce59c700 1 -- 172.21.15.77:0/2822680940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd01003a0 msgr2=0x7f8fd00fd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fce59c700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd01003a0 0x7f8fd00fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.477+0000 7f8fce59c700 1 -- 172.21.15.77:0/2822680940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc0005040 con 0x7f8fd00fda80 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.478+0000 7f8fce59c700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 0x7f8fd00fcf20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00093b0 tx=0x7f8fc0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.478+0000 7f8fbeffd700 1 -- 172.21.15.77:0/2822680940 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fc000e040 con 0x7f8fd00fda80 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.478+0000 7f8fbeffd700 1 -- 172.21.15.77:0/2822680940 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fc000bcc0 con 0x7f8fd00fda80 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.478+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd00fbe70 con 0x7f8fd00fda80 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.478+0000 7f8fbeffd700 1 -- 172.21.15.77:0/2822680940 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fc001c630 con 0x7f8fd00fda80 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.478+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd00fc3b0 con 0x7f8fd00fda80 2024-02-19T18:58:12.498 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.479+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd004bed0 con 0x7f8fd00fda80 2024-02-19T18:58:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.479+0000 7f8fbeffd700 1 -- 172.21.15.77:0/2822680940 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f8fc00139f0 con 0x7f8fd00fda80 2024-02-19T18:58:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.479+0000 7f8fbeffd700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8fb80692a0 0x7f8fb806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.480+0000 7f8fbeffd700 1 -- 172.21.15.77:0/2822680940 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8fc00882a0 con 0x7f8fd00fda80 2024-02-19T18:58:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.480+0000 7f8fcdd9b700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8fb80692a0 0x7f8fb806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:12.499 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.480+0000 7f8fcdd9b700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8fb80692a0 0x7f8fb806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00fccb0 tx=0x7f8fc4007cc0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:12.501 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.484+0000 7f8fbeffd700 1 -- 172.21.15.77:0/2822680940 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f8fc00549c0 con 0x7f8fd00fda80 2024-02-19T18:58:12.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.677+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fd0101000 con 0x7f8fb80692a0 2024-02-19T18:58:12.696 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.678+0000 7f8fbeffd700 1 -- 172.21.15.77:0/2822680940 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f8fd0101000 con 0x7f8fb80692a0 2024-02-19T18:58:12.696 INFO:teuthology.orchestra.run.smithi077.stdout:HOST ADDR LABELS STATUS 2024-02-19T18:58:12.696 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 172.21.15.77 2024-02-19T18:58:12.696 INFO:teuthology.orchestra.run.smithi077.stdout:smithi104 172.21.15.104 2024-02-19T18:58:12.696 INFO:teuthology.orchestra.run.smithi077.stdout:2 hosts in cluster 2024-02-19T18:58:12.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8fb80692a0 msgr2=0x7f8fb806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:12.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8fb80692a0 0x7f8fb806b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00fccb0 tx=0x7f8fc4007cc0 comp rx=0 tx=0).stop 2024-02-19T18:58:12.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 msgr2=0x7f8fd00fcf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:12.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 0x7f8fd00fcf20 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8fc00093b0 tx=0x7f8fc0009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:12.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 shutdown_connections 2024-02-19T18:58:12.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f8fb80692a0 0x7f8fb806b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:12.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f8fd00fda80 0x7f8fd00fcf20 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:12.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 --2- 172.21.15.77:0/2822680940 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f8fd01003a0 0x7f8fd00fd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:12.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 >> 172.21.15.77:0/2822680940 conn(0x7f8fd00f7520 msgr2=0x7f8fd00f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:12.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 shutdown_connections 2024-02-19T18:58:12.699 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:12.680+0000 7f8fd4b59700 1 -- 172.21.15.77:0/2822680940 wait complete. 2024-02-19T18:58:13.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:13 smithi077 ceph-mon[29247]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:13.377 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-19T18:58:13.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:13 smithi104 ceph-mon[42320]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:13.635 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:14.456 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:14 smithi077 ceph-mon[29247]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:14.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:14 smithi104 ceph-mon[42320]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 -- 172.21.15.77:0/2435135621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 msgr2=0x7fa77c0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 --2- 172.21.15.77:0/2435135621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 0x7fa77c0fd320 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fa778004940 tx=0x7fa778009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 -- 172.21.15.77:0/2435135621 shutdown_connections 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 --2- 172.21.15.77:0/2435135621 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa77c0fe0e0 0x7fa77c0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 --2- 172.21.15.77:0/2435135621 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 0x7fa77c0fd320 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 -- 172.21.15.77:0/2435135621 >> 172.21.15.77:0/2435135621 conn(0x7fa77c0f8620 msgr2=0x7fa77c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 -- 172.21.15.77:0/2435135621 shutdown_connections 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 -- 172.21.15.77:0/2435135621 wait complete. 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 Processor -- start 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.226+0000 7fa783b7a700 1 -- start start 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa783b7a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 0x7fa77c10d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:15.246 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa783b7a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa77c0fe0e0 0x7fa77c10deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa783b7a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c10e4b0 con 0x7fa77c0fcef0 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa783b7a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa77c10ed50 con 0x7fa77c0fe0e0 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781916700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 0x7fa77c10d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781916700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 0x7fa77c10d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58574/0 (socket says 172.21.15.77:58574) 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781916700 1 -- 172.21.15.77:0/2636017617 learned_addr learned my addr 172.21.15.77:0/2636017617 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781115700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa77c0fe0e0 0x7fa77c10deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781916700 1 -- 172.21.15.77:0/2636017617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa77c0fe0e0 msgr2=0x7fa77c10deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781916700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa77c0fe0e0 0x7fa77c10deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781916700 1 -- 172.21.15.77:0/2636017617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa778005040 con 0x7fa77c0fcef0 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781115700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa77c0fe0e0 0x7fa77c10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa781916700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 0x7fa77c10d970 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa77800b3b0 tx=0x7fa778009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa7727fc700 1 -- 172.21.15.77:0/2636017617 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa77800e040 con 0x7fa77c0fcef0 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa7727fc700 1 -- 172.21.15.77:0/2636017617 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa77800bcc0 con 0x7fa77c0fcef0 2024-02-19T18:58:15.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa7727fc700 1 -- 172.21.15.77:0/2636017617 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa77801c630 con 0x7fa77c0fcef0 2024-02-19T18:58:15.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.227+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa77c10efd0 con 0x7fa77c0fcef0 2024-02-19T18:58:15.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.228+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa77c10f510 con 0x7fa77c0fcef0 2024-02-19T18:58:15.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.228+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa77c04bed0 con 0x7fa77c0fcef0 2024-02-19T18:58:15.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.229+0000 7fa7727fc700 1 -- 172.21.15.77:0/2636017617 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa7780139f0 con 0x7fa77c0fcef0 2024-02-19T18:58:15.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.229+0000 7fa7727fc700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa768069370 0x7fa76806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:15.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.229+0000 7fa7727fc700 1 -- 172.21.15.77:0/2636017617 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa7780881c0 con 0x7fa77c0fcef0 2024-02-19T18:58:15.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.229+0000 7fa781115700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa768069370 0x7fa76806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:15.249 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.230+0000 7fa781115700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa768069370 0x7fa76806b820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009fd0 tx=0x7fa76c009ee0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:15.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.234+0000 7fa7727fc700 1 -- 172.21.15.77:0/2636017617 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa778058360 con 0x7fa77c0fcef0 2024-02-19T18:58:15.251 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:15 smithi077 ceph-mon[29247]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:15.443 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.425+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa77c102ac0 con 0x7fa768069370 2024-02-19T18:58:15.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.427+0000 7fa7727fc700 1 -- 172.21.15.77:0/2636017617 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7fa77c102ac0 con 0x7fa768069370 2024-02-19T18:58:15.445 INFO:teuthology.orchestra.run.smithi077.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-19T18:58:15.445 INFO:teuthology.orchestra.run.smithi077.stdout:smithi077 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-02-19T18:58:15.445 INFO:teuthology.orchestra.run.smithi077.stdout:smithi104 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NW400BGN 372G No 71s ago Has a FileSystem, LVM detected 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.428+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa768069370 msgr2=0x7fa76806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.428+0000 7fa783b7a700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa768069370 0x7fa76806b820 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa76c009fd0 tx=0x7fa76c009ee0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 msgr2=0x7fa77c10d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 0x7fa77c10d970 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fa77800b3b0 tx=0x7fa778009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 shutdown_connections 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa768069370 0x7fa76806b820 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa77c0fcef0 0x7fa77c10d970 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 --2- 172.21.15.77:0/2636017617 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa77c0fe0e0 0x7fa77c10deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:15.447 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 >> 172.21.15.77:0/2636017617 conn(0x7fa77c0f8620 msgr2=0x7fa77c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:15.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 shutdown_connections 2024-02-19T18:58:15.448 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:15.429+0000 7fa783b7a700 1 -- 172.21.15.77:0/2636017617 wait complete. 2024-02-19T18:58:15.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:15 smithi104 ceph-mon[42320]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:16.078 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-02-19T18:58:16.089 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:58:16.099 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:58:16.099 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-02-19T18:58:16.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:16 smithi077 ceph-mon[29247]: from='client.14504 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:16.356 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:16.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:16 smithi104 ceph-mon[42320]: from='client.14504 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:17.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:17 smithi077 ceph-mon[29247]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:17.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:17 smithi104 ceph-mon[42320]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:17.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.831+0000 7fa928a2f700 1 -- 172.21.15.77:0/2896828979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 msgr2=0x7fa9240fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:17.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.831+0000 7fa928a2f700 1 --2- 172.21.15.77:0/2896828979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 0x7fa9240fd420 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fa918004940 tx=0x7fa918009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:17.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 -- 172.21.15.77:0/2896828979 shutdown_connections 2024-02-19T18:58:17.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 --2- 172.21.15.77:0/2896828979 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 0x7fa9240fd420 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:17.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 --2- 172.21.15.77:0/2896828979 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9240fbe40 0x7fa9240fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:17.851 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 -- 172.21.15.77:0/2896828979 >> 172.21.15.77:0/2896828979 conn(0x7fa9240f7520 msgr2=0x7fa9240f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 -- 172.21.15.77:0/2896828979 shutdown_connections 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 -- 172.21.15.77:0/2896828979 wait complete. 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 Processor -- start 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 -- start start 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9240fbe40 0x7fa9240fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 0x7fa9240fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240fea10 con 0x7fa9240fcfb0 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa928a2f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9240feb80 con 0x7fa9240fbe40 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa921d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 0x7fa9240fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa921d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 0x7fa9240fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49484/0 (socket says 172.21.15.77:49484) 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.832+0000 7fa921d9b700 1 -- 172.21.15.77:0/3623071876 learned_addr learned my addr 172.21.15.77:0/3623071876 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:17.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa921d9b700 1 -- 172.21.15.77:0/3623071876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9240fbe40 msgr2=0x7fa9240fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa921d9b700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9240fbe40 0x7fa9240fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa921d9b700 1 -- 172.21.15.77:0/3623071876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918005040 con 0x7fa9240fcfb0 2024-02-19T18:58:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa921d9b700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 0x7fa9240fe380 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9180093b0 tx=0x7fa918009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa912ffd700 1 -- 172.21.15.77:0/3623071876 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa91800e040 con 0x7fa9240fcfb0 2024-02-19T18:58:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9240fee00 con 0x7fa9240fcfb0 2024-02-19T18:58:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa912ffd700 1 -- 172.21.15.77:0/3623071876 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa91800bcc0 con 0x7fa9240fcfb0 2024-02-19T18:58:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa912ffd700 1 -- 172.21.15.77:0/3623071876 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa91801c630 con 0x7fa9240fcfb0 2024-02-19T18:58:17.853 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.833+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa924115950 con 0x7fa9240fcfb0 2024-02-19T18:58:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.834+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92404bed0 con 0x7fa9240fcfb0 2024-02-19T18:58:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.834+0000 7fa912ffd700 1 -- 172.21.15.77:0/3623071876 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa9180139f0 con 0x7fa9240fcfb0 2024-02-19T18:58:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.835+0000 7fa912ffd700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa90c0692a0 0x7fa90c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.835+0000 7fa92259c700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa90c0692a0 0x7fa90c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.835+0000 7fa912ffd700 1 -- 172.21.15.77:0/3623071876 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa918088250 con 0x7fa9240fcfb0 2024-02-19T18:58:17.854 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.835+0000 7fa92259c700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa90c0692a0 0x7fa90c06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa914006fd0 tx=0x7fa914006db0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:17.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:17.839+0000 7fa912ffd700 1 -- 172.21.15.77:0/3623071876 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa918058350 con 0x7fa9240fcfb0 2024-02-19T18:58:18.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:18.052+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa924066660 con 0x7fa90c0692a0 2024-02-19T18:58:19.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:18 smithi077 ceph-mon[29247]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:19.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:18 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-19T18:58:19.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:18 smithi077 ceph-mon[29247]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:19.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:18 smithi104 ceph-mon[42320]: from='client.14508 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:58:19.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:18 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-02-19T18:58:19.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:18 smithi104 ceph-mon[42320]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:19.922 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.904+0000 7fa912ffd700 1 -- 172.21.15.77:0/3623071876 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa924066660 con 0x7fa90c0692a0 2024-02-19T18:58:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.907+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa90c0692a0 msgr2=0x7fa90c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.907+0000 7fa928a2f700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa90c0692a0 0x7fa90c06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa914006fd0 tx=0x7fa914006db0 comp rx=0 tx=0).stop 2024-02-19T18:58:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.907+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 msgr2=0x7fa9240fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:19.925 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.907+0000 7fa928a2f700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 0x7fa9240fe380 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa9180093b0 tx=0x7fa918009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:19.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.908+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 shutdown_connections 2024-02-19T18:58:19.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.908+0000 7fa928a2f700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa90c0692a0 0x7fa90c06b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:19.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.908+0000 7fa928a2f700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9240fbe40 0x7fa9240fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:19.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.908+0000 7fa928a2f700 1 --2- 172.21.15.77:0/3623071876 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9240fcfb0 0x7fa9240fe380 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:19.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.908+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 >> 172.21.15.77:0/3623071876 conn(0x7fa9240f7520 msgr2=0x7fa924065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:19.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.909+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 shutdown_connections 2024-02-19T18:58:19.927 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:19.909+0000 7fa928a2f700 1 -- 172.21.15.77:0/3623071876 wait complete. 2024-02-19T18:58:20.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-19T18:58:20.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:19 smithi077 ceph-mon[29247]: osdmap e42: 6 total, 6 up, 6 in 2024-02-19T18:58:20.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-19T18:58:20.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:20.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T18:58:19.884+0000 7fe140b47700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T18:58:20.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-02-19T18:58:20.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:19 smithi104 ceph-mon[42320]: osdmap e42: 6 total, 6 up, 6 in 2024-02-19T18:58:20.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-02-19T18:58:20.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:20.576 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-19T18:58:21.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-19T18:58:21.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: osdmap e43: 6 total, 6 up, 6 in 2024-02-19T18:58:21.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-19T18:58:21.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T18:58:21.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-19T18:58:21.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: osdmap e44: 6 total, 6 up, 6 in 2024-02-19T18:58:21.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-19T18:58:21.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: fsmap cephfs:0 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: Saving service mds.cephfs spec with placement count:4 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi077.ovassg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi077.ovassg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: Deploying daemon mds.cephfs.smithi077.ovassg on smithi077 2024-02-19T18:58:21.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:20 smithi077 ceph-mon[29247]: pgmap v175: 65 pgs: 3 creating+peering, 1 active+clean, 61 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: osdmap e43: 6 total, 6 up, 6 in 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: osdmap e44: 6 total, 6 up, 6 in 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: fsmap cephfs:0 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: Saving service mds.cephfs spec with placement count:4 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:21.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:21.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:21.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:21.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi077.ovassg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T18:58:21.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi077.ovassg", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T18:58:21.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:21.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: Deploying daemon mds.cephfs.smithi077.ovassg on smithi077 2024-02-19T18:58:21.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:20 smithi104 ceph-mon[42320]: pgmap v175: 65 pgs: 3 creating+peering, 1 active+clean, 61 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:21.367 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:22.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:21 smithi077 ceph-mon[29247]: osdmap e45: 6 total, 6 up, 6 in 2024-02-19T18:58:22.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:21 smithi104 ceph-mon[42320]: osdmap e45: 6 total, 6 up, 6 in 2024-02-19T18:58:23.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:22 smithi077 ceph-mon[29247]: osdmap e46: 6 total, 6 up, 6 in 2024-02-19T18:58:23.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:22 smithi077 ceph-mon[29247]: pgmap v178: 65 pgs: 22 active+clean, 3 creating+peering, 40 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:23.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:22 smithi104 ceph-mon[42320]: osdmap e46: 6 total, 6 up, 6 in 2024-02-19T18:58:23.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:22 smithi104 ceph-mon[42320]: pgmap v178: 65 pgs: 22 active+clean, 3 creating+peering, 40 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:23.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.517+0000 7ff533cba700 1 -- 172.21.15.77:0/229709554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fe0e0 msgr2=0x7ff52c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.517+0000 7ff533cba700 1 --2- 172.21.15.77:0/229709554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fe0e0 0x7ff52c0fe550 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff51c004940 tx=0x7ff51c009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.517+0000 7ff533cba700 1 -- 172.21.15.77:0/229709554 shutdown_connections 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.517+0000 7ff533cba700 1 --2- 172.21.15.77:0/229709554 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fe0e0 0x7ff52c0fe550 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.517+0000 7ff533cba700 1 --2- 172.21.15.77:0/229709554 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff52c0fcef0 0x7ff52c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.517+0000 7ff533cba700 1 -- 172.21.15.77:0/229709554 >> 172.21.15.77:0/229709554 conn(0x7ff52c0f8620 msgr2=0x7ff52c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.517+0000 7ff533cba700 1 -- 172.21.15.77:0/229709554 shutdown_connections 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.517+0000 7ff533cba700 1 -- 172.21.15.77:0/229709554 wait complete. 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff533cba700 1 Processor -- start 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff533cba700 1 -- start start 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff533cba700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fcef0 0x7ff52c10d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff533cba700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff52c0fe0e0 0x7ff52c10de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:23.948 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff533cba700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c10e430 con 0x7ff52c0fcef0 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff533cba700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c10ecd0 con 0x7ff52c0fe0e0 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff531a56700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fcef0 0x7ff52c10d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff531a56700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fcef0 0x7ff52c10d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49500/0 (socket says 172.21.15.77:49500) 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff531a56700 1 -- 172.21.15.77:0/3878016312 learned_addr learned my addr 172.21.15.77:0/3878016312 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff531255700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff52c0fe0e0 0x7ff52c10de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff531a56700 1 -- 172.21.15.77:0/3878016312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff52c0fe0e0 msgr2=0x7ff52c10de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff531a56700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff52c0fe0e0 0x7ff52c10de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stdout:e2 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T18:58:23.949 INFO:teuthology.orchestra.run.smithi077.stdout:legacy client fscid: 1 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:Filesystem 'cephfs' (1) 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:fs_name cephfs 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 2 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-02-19T18:58:19.884992+0000 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:modified 2024-02-19T18:58:19.885062+0000 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:tableserver 0 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:root 0 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:session_timeout 60 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:session_autoclose 300 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:max_file_size 1099511627776 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:required_client_features {} 2024-02-19T18:58:23.950 INFO:teuthology.orchestra.run.smithi077.stdout:last_failure 0 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:last_failure_osd_epoch 0 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:max_mds 1 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:in 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:up {} 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:failed 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:damaged 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:stopped 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:data_pools [3] 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:metadata_pool 2 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:inline_data disabled 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:balancer 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:standby_count_wanted 0 2024-02-19T18:58:23.951 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.518+0000 7ff531a56700 1 -- 172.21.15.77:0/3878016312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c005040 con 0x7ff52c0fcef0 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.519+0000 7ff531a56700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fcef0 0x7ff52c10d8f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff52800eb70 tx=0x7ff52800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.519+0000 7ff5227fc700 1 -- 172.21.15.77:0/3878016312 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff528007cc0 con 0x7ff52c0fcef0 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.519+0000 7ff5227fc700 1 -- 172.21.15.77:0/3878016312 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff52800f040 con 0x7ff52c0fcef0 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.519+0000 7ff5227fc700 1 -- 172.21.15.77:0/3878016312 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff52800cbd0 con 0x7ff52c0fcef0 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.519+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52c10ef50 con 0x7ff52c0fcef0 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.519+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c10f490 con 0x7ff52c0fcef0 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.520+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c04bed0 con 0x7ff52c0fcef0 2024-02-19T18:58:23.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.520+0000 7ff5227fc700 1 -- 172.21.15.77:0/3878016312 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff52800cd30 con 0x7ff52c0fcef0 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.520+0000 7ff5227fc700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff5180692a0 0x7ff51806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.521+0000 7ff531255700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff5180692a0 0x7ff51806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.521+0000 7ff5227fc700 1 -- 172.21.15.77:0/3878016312 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff528086be0 con 0x7ff52c0fcef0 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.521+0000 7ff531255700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff5180692a0 0x7ff51806b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009b30 tx=0x7ff51c009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.525+0000 7ff5227fc700 1 -- 172.21.15.77:0/3878016312 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff52804dbd0 con 0x7ff52c0fcef0 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.757+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff52c10f8c0 con 0x7ff52c0fcef0 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.758+0000 7ff5227fc700 1 -- 172.21.15.77:0/3878016312 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7ff5280567c0 con 0x7ff52c0fcef0 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff5180692a0 msgr2=0x7ff51806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff5180692a0 0x7ff51806b750 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009b30 tx=0x7ff51c009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fcef0 msgr2=0x7ff52c10d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fcef0 0x7ff52c10d8f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff52800eb70 tx=0x7ff52800b040 comp rx=0 tx=0).stop 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 shutdown_connections 2024-02-19T18:58:23.953 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff5180692a0 0x7ff51806b750 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff52c0fcef0 0x7ff52c10d8f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 --2- 172.21.15.77:0/3878016312 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff52c0fe0e0 0x7ff52c10de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:23.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 >> 172.21.15.77:0/3878016312 conn(0x7ff52c0f8620 msgr2=0x7ff52c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:23.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 shutdown_connections 2024-02-19T18:58:23.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:23.760+0000 7ff533cba700 1 -- 172.21.15.77:0/3878016312 wait complete. 2024-02-19T18:58:23.954 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 2 2024-02-19T18:58:24.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:23 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3878016312' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T18:58:24.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:23 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:24.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:23 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:24.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:23 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:24.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:23 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.rudtoq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T18:58:24.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:23 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.rudtoq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T18:58:24.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:23 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:24.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:23 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3878016312' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T18:58:24.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:23 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:24.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:23 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:24.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:23 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:24.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:23 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.rudtoq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T18:58:24.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:23 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.rudtoq", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T18:58:24.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:23 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:24.633 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:58:24.643 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:58:24.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-02-19T18:58:24.899 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: Deploying daemon mds.cephfs.smithi104.rudtoq on smithi104 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.77:6826/3246598933,v1:172.21.15.77:6827/3246598933] up:boot 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: daemon mds.cephfs.smithi077.ovassg assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: Cluster is now healthy 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: fsmap cephfs:0 1 up:standby 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.ovassg"}]: dispatch 2024-02-19T18:58:25.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:creating} 2024-02-19T18:58:25.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: daemon mds.cephfs.smithi077.ovassg is now active in filesystem cephfs as rank 0 2024-02-19T18:58:25.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:24 smithi077 ceph-mon[29247]: pgmap v179: 65 pgs: 56 active+clean, 3 creating+peering, 6 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: Deploying daemon mds.cephfs.smithi104.rudtoq on smithi104 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.77:6826/3246598933,v1:172.21.15.77:6827/3246598933] up:boot 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: daemon mds.cephfs.smithi077.ovassg assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: Cluster is now healthy 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: fsmap cephfs:0 1 up:standby 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.ovassg"}]: dispatch 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:creating} 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: daemon mds.cephfs.smithi077.ovassg is now active in filesystem cephfs as rank 0 2024-02-19T18:58:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:24 smithi104 ceph-mon[42320]: pgmap v179: 65 pgs: 56 active+clean, 3 creating+peering, 6 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:26.287 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:25 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.77:6826/3246598933,v1:172.21.15.77:6827/3246598933] up:active 2024-02-19T18:58:26.288 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:25 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 2024-02-19T18:58:26.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:25 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.77:6826/3246598933,v1:172.21.15.77:6827/3246598933] up:active 2024-02-19T18:58:26.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:25 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.690+0000 7ff6435d7700 1 -- 172.21.15.77:0/2769906698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 msgr2=0x7ff63c0fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.690+0000 7ff6435d7700 1 --2- 172.21.15.77:0/2769906698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 0x7ff63c0fd430 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff62c004940 tx=0x7ff62c009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.690+0000 7ff6435d7700 1 -- 172.21.15.77:0/2769906698 shutdown_connections 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.690+0000 7ff6435d7700 1 --2- 172.21.15.77:0/2769906698 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 0x7ff63c0fd430 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.690+0000 7ff6435d7700 1 --2- 172.21.15.77:0/2769906698 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff63c0fbe50 0x7ff63c0fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.690+0000 7ff6435d7700 1 -- 172.21.15.77:0/2769906698 >> 172.21.15.77:0/2769906698 conn(0x7ff63c0f7510 msgr2=0x7ff63c0f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.691+0000 7ff6435d7700 1 -- 172.21.15.77:0/2769906698 shutdown_connections 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.691+0000 7ff6435d7700 1 -- 172.21.15.77:0/2769906698 wait complete. 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.691+0000 7ff6435d7700 1 Processor -- start 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.691+0000 7ff6435d7700 1 -- start start 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.691+0000 7ff6435d7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff63c0fbe50 0x7ff63c110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:26.710 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.691+0000 7ff6435d7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 0x7ff63c112bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.691+0000 7ff6435d7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c066cf0 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.691+0000 7ff6435d7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff63c066e30 con 0x7ff63c0fbe50 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff640b72700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 0x7ff63c112bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff640b72700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 0x7ff63c112bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55332/0 (socket says 172.21.15.77:55332) 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff640b72700 1 -- 172.21.15.77:0/3609516415 learned_addr learned my addr 172.21.15.77:0/3609516415 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff641373700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff63c0fbe50 0x7ff63c110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff640b72700 1 -- 172.21.15.77:0/3609516415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff63c0fbe50 msgr2=0x7ff63c110670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff640b72700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff63c0fbe50 0x7ff63c110670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff640b72700 1 -- 172.21.15.77:0/3609516415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c005040 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.711 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff640b72700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 0x7ff63c112bc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff62c0093b0 tx=0x7ff62c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:26.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff631ffb700 1 -- 172.21.15.77:0/3609516415 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff62c00e040 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63c1131c0 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff631ffb700 1 -- 172.21.15.77:0/3609516415 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff62c00bcc0 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff631ffb700 1 -- 172.21.15.77:0/3609516415 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff62c01c630 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.712 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.692+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63c113700 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.693+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63c04bed0 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.694+0000 7ff631ffb700 1 -- 172.21.15.77:0/3609516415 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff62c013980 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.694+0000 7ff631ffb700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff6280692a0 0x7ff62806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.695+0000 7ff631ffb700 1 -- 172.21.15.77:0/3609516415 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff62c088520 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.695+0000 7ff641373700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff6280692a0 0x7ff62806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:26.713 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.695+0000 7ff641373700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff6280692a0 0x7ff62806b750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6380041d0 tx=0x7ff6380043c0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:26.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.699+0000 7ff631ffb700 1 -- 172.21.15.77:0/3609516415 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff62c054850 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.945+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7ff63c113350 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.972+0000 7ff631ffb700 1 -- 172.21.15.77:0/3609516415 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff63c113350 con 0x7ff63c0fcfc0 2024-02-19T18:58:26.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff6280692a0 msgr2=0x7ff62806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff6280692a0 0x7ff62806b750 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff6380041d0 tx=0x7ff6380043c0 comp rx=0 tx=0).stop 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 msgr2=0x7ff63c112bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 0x7ff63c112bc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff62c0093b0 tx=0x7ff62c009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 shutdown_connections 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff6280692a0 0x7ff62806b750 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff63c0fbe50 0x7ff63c110670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 --2- 172.21.15.77:0/3609516415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff63c0fcfc0 0x7ff63c112bc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.974+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 >> 172.21.15.77:0/3609516415 conn(0x7ff63c0f7510 msgr2=0x7ff63c100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.975+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 shutdown_connections 2024-02-19T18:58:26.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:26.975+0000 7ff6435d7700 1 -- 172.21.15.77:0/3609516415 wait complete. 2024-02-19T18:58:27.688 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-19T18:58:27.688 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:27.688 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi077.jmjalp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi077.jmjalp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: Deploying daemon mds.cephfs.smithi077.jmjalp on smithi077 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3609516415' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.104:6824/2019831719,v1:172.21.15.104:6825/2019831719] up:boot 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3609516415' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 1 up:standby 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.rudtoq"}]: dispatch 2024-02-19T18:58:27.689 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:27 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 1 up:standby 2024-02-19T18:58:27.694 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: pgmap v180: 65 pgs: 65 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-02-19T18:58:27.694 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:27.694 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi077.jmjalp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi077.jmjalp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: Deploying daemon mds.cephfs.smithi077.jmjalp on smithi077 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3609516415' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.104:6824/2019831719,v1:172.21.15.104:6825/2019831719] up:boot 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3609516415' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 1 up:standby 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.rudtoq"}]: dispatch 2024-02-19T18:58:27.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:27 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 1 up:standby 2024-02-19T18:58:28.582 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:58:28.593 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:58:28.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay false' 2024-02-19T18:58:29.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:28 smithi077 ceph-mon[29247]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-02-19T18:58:29.100 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:29.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:28 smithi104 ceph-mon[42320]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.wpcqst", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.wpcqst", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: Deploying daemon mds.cephfs.smithi104.wpcqst on smithi104 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.77:6828/3630854907,v1:172.21.15.77:6829/3630854907] up:boot 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 2 up:standby 2024-02-19T18:58:30.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.jmjalp"}]: dispatch 2024-02-19T18:58:30.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 -- 172.21.15.77:0/378874231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fcf30 msgr2=0x7fb9440fd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:30.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 --2- 172.21.15.77:0/378874231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fcf30 0x7fb9440fd340 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fb938004940 tx=0x7fb938009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:30.488 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 -- 172.21.15.77:0/378874231 shutdown_connections 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 --2- 172.21.15.77:0/378874231 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9440fe1e0 0x7fb9440fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 --2- 172.21.15.77:0/378874231 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fcf30 0x7fb9440fd340 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 -- 172.21.15.77:0/378874231 >> 172.21.15.77:0/378874231 conn(0x7fb9440f86b0 msgr2=0x7fb9440faac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 -- 172.21.15.77:0/378874231 shutdown_connections 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 -- 172.21.15.77:0/378874231 wait complete. 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 Processor -- start 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.469+0000 7fb94aefc700 1 -- start start 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb94aefc700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9440fcf30 0x7fb944071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb94aefc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fe1e0 0x7fb9440702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb94aefc700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb944070970 con 0x7fb9440fe1e0 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb94aefc700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb944070ae0 con 0x7fb9440fcf30 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb943fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fe1e0 0x7fb9440702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb943fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fe1e0 0x7fb9440702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55360/0 (socket says 172.21.15.77:55360) 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb943fff700 1 -- 172.21.15.77:0/256268853 learned_addr learned my addr 172.21.15.77:0/256268853 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb948c98700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9440fcf30 0x7fb944071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb943fff700 1 -- 172.21.15.77:0/256268853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9440fcf30 msgr2=0x7fb944071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:30.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb943fff700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9440fcf30 0x7fb944071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:30.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb943fff700 1 -- 172.21.15.77:0/256268853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb938005040 con 0x7fb9440fe1e0 2024-02-19T18:58:30.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb948c98700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9440fcf30 0x7fb944071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:58:30.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb943fff700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fe1e0 0x7fb9440702e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb93000b790 tx=0x7fb930009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:30.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb9417fa700 1 -- 172.21.15.77:0/256268853 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb93000f770 con 0x7fb9440fe1e0 2024-02-19T18:58:30.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb9417fa700 1 -- 172.21.15.77:0/256268853 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb93000fda0 con 0x7fb9440fe1e0 2024-02-19T18:58:30.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb944070dc0 con 0x7fb9440fe1e0 2024-02-19T18:58:30.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb9417fa700 1 -- 172.21.15.77:0/256268853 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb93000d310 con 0x7fb9440fe1e0 2024-02-19T18:58:30.490 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.470+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb944071140 con 0x7fb9440fe1e0 2024-02-19T18:58:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.471+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb94404bed0 con 0x7fb9440fe1e0 2024-02-19T18:58:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.472+0000 7fb9417fa700 1 -- 172.21.15.77:0/256268853 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb930010420 con 0x7fb9440fe1e0 2024-02-19T18:58:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.472+0000 7fb9417fa700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb9340692a0 0x7fb93406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.472+0000 7fb9417fa700 1 -- 172.21.15.77:0/256268853 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb9300871b0 con 0x7fb9440fe1e0 2024-02-19T18:58:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.472+0000 7fb948c98700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb9340692a0 0x7fb93406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:30.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.473+0000 7fb948c98700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb9340692a0 0x7fb93406b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb938009fd0 tx=0x7fb938009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:30.494 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.476+0000 7fb9417fa700 1 -- 172.21.15.77:0/256268853 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb930056f70 con 0x7fb9440fe1e0 2024-02-19T18:58:30.502 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:30.502 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:30.502 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:30.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.wpcqst", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-19T18:58:30.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi104.wpcqst", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-19T18:58:30.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:30.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: Deploying daemon mds.cephfs.smithi104.wpcqst on smithi104 2024-02-19T18:58:30.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.77:6828/3630854907,v1:172.21.15.77:6829/3630854907] up:boot 2024-02-19T18:58:30.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 2 up:standby 2024-02-19T18:58:30.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.jmjalp"}]: dispatch 2024-02-19T18:58:30.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:30.703+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb944115ec0 con 0x7fb9440fe1e0 2024-02-19T18:58:31.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.100+0000 7fb9417fa700 1 -- 172.21.15.77:0/256268853 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb930056b00 con 0x7fb9440fe1e0 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.102+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb9340692a0 msgr2=0x7fb93406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.102+0000 7fb94aefc700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb9340692a0 0x7fb93406b750 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb938009fd0 tx=0x7fb938009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.102+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fe1e0 msgr2=0x7fb9440702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.102+0000 7fb94aefc700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fe1e0 0x7fb9440702e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb93000b790 tx=0x7fb930009250 comp rx=0 tx=0).stop 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.103+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 shutdown_connections 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.103+0000 7fb94aefc700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb9340692a0 0x7fb93406b750 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.103+0000 7fb94aefc700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb9440fcf30 0x7fb944071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.103+0000 7fb94aefc700 1 --2- 172.21.15.77:0/256268853 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb9440fe1e0 0x7fb9440702e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.103+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 >> 172.21.15.77:0/256268853 conn(0x7fb9440f86b0 msgr2=0x7fb944101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.103+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 shutdown_connections 2024-02-19T18:58:31.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:31.103+0000 7fb94aefc700 1 -- 172.21.15.77:0/256268853 wait complete. 2024-02-19T18:58:31.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:31 smithi104 ceph-mon[42320]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-02-19T18:58:31.171 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:31 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/256268853' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-02-19T18:58:31.406 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:31 smithi077 ceph-mon[29247]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s wr, 4 op/s 2024-02-19T18:58:31.406 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:31 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/256268853' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]: dispatch 2024-02-19T18:58:31.689 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:58:31.698 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:58:31.698 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-02-19T18:58:31.972 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:32.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:32 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/256268853' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-02-19T18:58:32.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:32 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 2 up:standby 2024-02-19T18:58:32.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:32 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:32 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:32 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:32 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:32 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:32 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:32.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:32 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/256268853' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]': finished 2024-02-19T18:58:32.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:32 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 2 up:standby 2024-02-19T18:58:32.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:32 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:32 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:32 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:32 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:32 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:32.444 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:32 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:33.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:33 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.104:6826/2946362025,v1:172.21.15.104:6827/2946362025] up:boot 2024-02-19T18:58:33.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:33 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.77:6826/3246598933,v1:172.21.15.77:6827/3246598933] up:active 2024-02-19T18:58:33.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:33 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 3 up:standby 2024-02-19T18:58:33.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.wpcqst"}]: dispatch 2024-02-19T18:58:33.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:33 smithi104 ceph-mon[42320]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-19T18:58:33.320 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:33 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.104:6826/2946362025,v1:172.21.15.104:6827/2946362025] up:boot 2024-02-19T18:58:33.320 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:33 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.77:6826/3246598933,v1:172.21.15.77:6827/3246598933] up:active 2024-02-19T18:58:33.320 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:33 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 3 up:standby 2024-02-19T18:58:33.320 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.wpcqst"}]: dispatch 2024-02-19T18:58:33.320 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:33 smithi077 ceph-mon[29247]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-02-19T18:58:33.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 -- 172.21.15.77:0/497570291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fcef0 msgr2=0x7fe9180fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:33.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 --2- 172.21.15.77:0/497570291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fcef0 0x7fe9180fd320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe908004940 tx=0x7fe908009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:33.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 -- 172.21.15.77:0/497570291 shutdown_connections 2024-02-19T18:58:33.545 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 --2- 172.21.15.77:0/497570291 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9180fe0e0 0x7fe9180fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:33.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 --2- 172.21.15.77:0/497570291 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fcef0 0x7fe9180fd320 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:33.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 -- 172.21.15.77:0/497570291 >> 172.21.15.77:0/497570291 conn(0x7fe9180f8620 msgr2=0x7fe9180faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:33.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 -- 172.21.15.77:0/497570291 shutdown_connections 2024-02-19T18:58:33.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 -- 172.21.15.77:0/497570291 wait complete. 2024-02-19T18:58:33.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 Processor -- start 2024-02-19T18:58:33.546 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.528+0000 7fe91dc24700 1 -- start start 2024-02-19T18:58:33.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe91dc24700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9180fcef0 0x7fe91810d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:33.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe91dc24700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fe0e0 0x7fe91810de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:33.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe91dc24700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91810e450 con 0x7fe9180fe0e0 2024-02-19T18:58:33.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe91dc24700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe91810ecf0 con 0x7fe9180fcef0 2024-02-19T18:58:33.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe916ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fe0e0 0x7fe91810de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:33.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe916ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fe0e0 0x7fe91810de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:55370/0 (socket says 172.21.15.77:55370) 2024-02-19T18:58:33.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe9177fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9180fcef0 0x7fe91810d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe916ffd700 1 -- 172.21.15.77:0/1051575983 learned_addr learned my addr 172.21.15.77:0/1051575983 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe9177fe700 1 -- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fe0e0 msgr2=0x7fe91810de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe9177fe700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fe0e0 0x7fe91810de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe9177fe700 1 -- 172.21.15.77:0/1051575983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe908005040 con 0x7fe9180fcef0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe9177fe700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9180fcef0 0x7fe91810d910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe908000c00 tx=0x7fe908009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe8fffff700 1 -- 172.21.15.77:0/1051575983 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe90800e070 con 0x7fe9180fcef0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe91810ef70 con 0x7fe9180fcef0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe8fffff700 1 -- 172.21.15.77:0/1051575983 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe90801c400 con 0x7fe9180fcef0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe8fffff700 1 -- 172.21.15.77:0/1051575983 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe90800b650 con 0x7fe9180fcef0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.529+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91810f450 con 0x7fe9180fcef0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.530+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91804bed0 con 0x7fe9180fcef0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.531+0000 7fe8fffff700 1 -- 172.21.15.77:0/1051575983 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe90801ca50 con 0x7fe9180fcef0 2024-02-19T18:58:33.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.531+0000 7fe8fffff700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe9000692b0 0x7fe90006b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:33.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.531+0000 7fe916ffd700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe9000692b0 0x7fe90006b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:33.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.531+0000 7fe8fffff700 1 -- 172.21.15.77:0/1051575983 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe9080887c0 con 0x7fe9180fcef0 2024-02-19T18:58:33.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.532+0000 7fe916ffd700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe9000692b0 0x7fe90006b760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe90c004af0 tx=0x7fe90c00b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:33.553 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.536+0000 7fe8fffff700 1 -- 172.21.15.77:0/1051575983 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe9080585a0 con 0x7fe9180fcef0 2024-02-19T18:58:33.786 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.768+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe918105a50 con 0x7fe9180fcef0 2024-02-19T18:58:33.891 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.873+0000 7fe8fffff700 1 -- 172.21.15.77:0/1051575983 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v11) v1 ==== 168+0+0 (secure 0 0 0) 0x7fe908058130 con 0x7fe9180fcef0 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe9000692b0 msgr2=0x7fe90006b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe9000692b0 0x7fe90006b760 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe90c004af0 tx=0x7fe90c00b040 comp rx=0 tx=0).stop 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9180fcef0 msgr2=0x7fe91810d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9180fcef0 0x7fe91810d910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe908000c00 tx=0x7fe908009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 shutdown_connections 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe9000692b0 0x7fe90006b760 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe9180fcef0 0x7fe91810d910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 --2- 172.21.15.77:0/1051575983 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe9180fe0e0 0x7fe91810de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 >> 172.21.15.77:0/1051575983 conn(0x7fe9180f8620 msgr2=0x7fe9181013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 shutdown_connections 2024-02-19T18:58:33.893 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:33.875+0000 7fe91dc24700 1 -- 172.21.15.77:0/1051575983 wait complete. 2024-02-19T18:58:33.894 INFO:teuthology.orchestra.run.smithi077.stderr:inline data enabled 2024-02-19T18:58:34.739 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-19T18:58:34.750 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:58:34.750 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1051575983' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.77:6828/3630854907,v1:172.21.15.77:6829/3630854907] up:standby 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 3 up:standby 2024-02-19T18:58:34.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-19T18:58:34.777 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:34.777 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:34.777 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:35.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:35.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:35.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1051575983' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T18:58:35.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-19T18:58:35.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-02-19T18:58:35.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.77:6828/3630854907,v1:172.21.15.77:6829/3630854907] up:standby 2024-02-19T18:58:35.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-02-19T18:58:35.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 3 up:standby 2024-02-19T18:58:35.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-19T18:58:35.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:35.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:35.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:35.518 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:36.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:36 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.104:6824/2019831719,v1:172.21.15.104:6825/2019831719] up:standby 2024-02-19T18:58:36.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:36 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 3 up:standby 2024-02-19T18:58:36.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:36 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.104:6824/2019831719,v1:172.21.15.104:6825/2019831719] up:standby 2024-02-19T18:58:36.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:36 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 3 up:standby 2024-02-19T18:58:37.304 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:37 smithi077 ceph-mon[29247]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-19T18:58:37.304 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:37 smithi077 ceph-mon[29247]: mds.? [v2:172.21.15.104:6826/2946362025,v1:172.21.15.104:6827/2946362025] up:standby 2024-02-19T18:58:37.304 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:37 smithi077 ceph-mon[29247]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 3 up:standby 2024-02-19T18:58:37.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:37 smithi104 ceph-mon[42320]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-02-19T18:58:37.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:37 smithi104 ceph-mon[42320]: mds.? [v2:172.21.15.104:6826/2946362025,v1:172.21.15.104:6827/2946362025] up:standby 2024-02-19T18:58:37.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:37 smithi104 ceph-mon[42320]: fsmap cephfs:1 {0=cephfs.smithi077.ovassg=up:active} 3 up:standby 2024-02-19T18:58:37.807 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.787+0000 7faf4a73b700 1 -- 172.21.15.77:0/1181122263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 msgr2=0x7faf440fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.787+0000 7faf4a73b700 1 --2- 172.21.15.77:0/1181122263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 0x7faf440fc250 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7faf34004940 tx=0x7faf34009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.787+0000 7faf4a73b700 1 -- 172.21.15.77:0/1181122263 shutdown_connections 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.787+0000 7faf4a73b700 1 --2- 172.21.15.77:0/1181122263 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf440fcfb0 0x7faf440fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.787+0000 7faf4a73b700 1 --2- 172.21.15.77:0/1181122263 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 0x7faf440fc250 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 -- 172.21.15.77:0/1181122263 >> 172.21.15.77:0/1181122263 conn(0x7faf440f7520 msgr2=0x7faf440f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 -- 172.21.15.77:0/1181122263 shutdown_connections 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 -- 172.21.15.77:0/1181122263 wait complete. 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 Processor -- start 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 -- start start 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 0x7faf4410c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:37.808 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf440fcfb0 0x7faf4410cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4410d420 con 0x7faf440fbe40 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.788+0000 7faf4a73b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4410dc60 con 0x7faf440fcfb0 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf43fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 0x7faf4410c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf43fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 0x7faf4410c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35066/0 (socket says 172.21.15.77:35066) 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf43fff700 1 -- 172.21.15.77:0/3381494609 learned_addr learned my addr 172.21.15.77:0/3381494609 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf437fe700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf440fcfb0 0x7faf4410cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf43fff700 1 -- 172.21.15.77:0/3381494609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf440fcfb0 msgr2=0x7faf4410cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf43fff700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf440fcfb0 0x7faf4410cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf43fff700 1 -- 172.21.15.77:0/3381494609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf34005040 con 0x7faf440fbe40 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf43fff700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 0x7faf4410c850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7faf340093b0 tx=0x7faf34009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf40ff9700 1 -- 172.21.15.77:0/3381494609 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf3400e040 con 0x7faf440fbe40 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf4410dee0 con 0x7faf440fbe40 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf40ff9700 1 -- 172.21.15.77:0/3381494609 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf3400bcc0 con 0x7faf440fbe40 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf40ff9700 1 -- 172.21.15.77:0/3381494609 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf3401c630 con 0x7faf440fbe40 2024-02-19T18:58:37.809 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.789+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf4410e420 con 0x7faf440fbe40 2024-02-19T18:58:37.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.790+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf4404bed0 con 0x7faf440fbe40 2024-02-19T18:58:37.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.791+0000 7faf40ff9700 1 -- 172.21.15.77:0/3381494609 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7faf340139f0 con 0x7faf440fbe40 2024-02-19T18:58:37.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.791+0000 7faf40ff9700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7faf2c0692a0 0x7faf2c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:37.810 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.791+0000 7faf437fe700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7faf2c0692a0 0x7faf2c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:37.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.791+0000 7faf40ff9700 1 -- 172.21.15.77:0/3381494609 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7faf34088590 con 0x7faf440fbe40 2024-02-19T18:58:37.811 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.791+0000 7faf437fe700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7faf2c0692a0 0x7faf2c06b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faf38006fd0 tx=0x7faf38008040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:37.813 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:37.795+0000 7faf40ff9700 1 -- 172.21.15.77:0/3381494609 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7faf34058350 con 0x7faf440fbe40 2024-02-19T18:58:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.036+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7faf4410e5b0 con 0x7faf440fbe40 2024-02-19T18:58:38.055 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.037+0000 7faf40ff9700 1 -- 172.21.15.77:0/3381494609 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 13 v13) v1 ==== 76+0+1810 (secure 0 0 0) 0x7faf34057ee0 con 0x7faf440fbe40 2024-02-19T18:58:38.055 INFO:teuthology.orchestra.run.smithi077.stdout:e13 2024-02-19T18:58:38.055 INFO:teuthology.orchestra.run.smithi077.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout:legacy client fscid: 1 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout:Filesystem 'cephfs' (1) 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout:fs_name cephfs 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 11 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-02-19T18:58:19.884992+0000 2024-02-19T18:58:38.056 INFO:teuthology.orchestra.run.smithi077.stdout:modified 2024-02-19T18:58:33.870342+0000 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:tableserver 0 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:root 0 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:session_timeout 60 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:session_autoclose 300 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:max_file_size 1099511627776 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:required_client_features {} 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:last_failure 0 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:last_failure_osd_epoch 0 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:max_mds 1 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:in 0 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:up {0=14516} 2024-02-19T18:58:38.057 INFO:teuthology.orchestra.run.smithi077.stdout:failed 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:damaged 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:stopped 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:data_pools [3] 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:metadata_pool 2 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:inline_data enabled 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:balancer 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:standby_count_wanted 1 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.cephfs.smithi077.ovassg{0:14516} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.77:6826/3246598933,v1:172.21.15.77:6827/3246598933] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout:Standby daemons: 2024-02-19T18:58:38.058 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:38.059 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.cephfs.smithi077.jmjalp{-1:14528} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.77:6828/3630854907,v1:172.21.15.77:6829/3630854907] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T18:58:38.059 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.cephfs.smithi104.rudtoq{-1:24253} state up:standby seq 3 join_fscid=1 addr [v2:172.21.15.104:6824/2019831719,v1:172.21.15.104:6825/2019831719] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T18:58:38.059 INFO:teuthology.orchestra.run.smithi077.stdout:[mds.cephfs.smithi104.wpcqst{-1:24263} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.104:6826/2946362025,v1:172.21.15.104:6827/2946362025] compat {c=[1],r=[1],i=[7ff]}] 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7faf2c0692a0 msgr2=0x7faf2c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7faf2c0692a0 0x7faf2c06b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7faf38006fd0 tx=0x7faf38008040 comp rx=0 tx=0).stop 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 msgr2=0x7faf4410c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 0x7faf4410c850 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7faf340093b0 tx=0x7faf34009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 shutdown_connections 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7faf2c0692a0 0x7faf2c06b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faf440fbe40 0x7faf4410c850 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 --2- 172.21.15.77:0/3381494609 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faf440fcfb0 0x7faf4410cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 >> 172.21.15.77:0/3381494609 conn(0x7faf440f7520 msgr2=0x7faf44100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:38.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 shutdown_connections 2024-02-19T18:58:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:38.039+0000 7faf4a73b700 1 -- 172.21.15.77:0/3381494609 wait complete. 2024-02-19T18:58:38.061 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-02-19T18:58:38.671 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-02-19T18:58:39.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:38 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3381494609' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T18:58:39.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:38 smithi077 ceph-mon[29247]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 2 op/s 2024-02-19T18:58:39.062 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:38 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3381494609' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-19T18:58:39.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:38 smithi104 ceph-mon[42320]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 2 op/s 2024-02-19T18:58:39.088 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:41.207 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.187+0000 7f0e88399700 1 -- 172.21.15.77:0/2733786326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fcfc0 msgr2=0x7f0e800fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.187+0000 7f0e88399700 1 --2- 172.21.15.77:0/2733786326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fcfc0 0x7f0e800fd430 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0e70004940 tx=0x7f0e70009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.187+0000 7f0e88399700 1 -- 172.21.15.77:0/2733786326 shutdown_connections 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.187+0000 7f0e88399700 1 --2- 172.21.15.77:0/2733786326 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fcfc0 0x7f0e800fd430 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.187+0000 7f0e88399700 1 --2- 172.21.15.77:0/2733786326 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e800fbe50 0x7f0e800fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.187+0000 7f0e88399700 1 -- 172.21.15.77:0/2733786326 >> 172.21.15.77:0/2733786326 conn(0x7f0e800f7510 msgr2=0x7f0e800f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.187+0000 7f0e88399700 1 -- 172.21.15.77:0/2733786326 shutdown_connections 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.187+0000 7f0e88399700 1 -- 172.21.15.77:0/2733786326 wait complete. 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e88399700 1 Processor -- start 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e88399700 1 -- start start 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e88399700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fbe50 0x7f0e800fff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e88399700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e800fcfc0 0x7f0e800fe390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e88399700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e800fe990 con 0x7f0e800fbe50 2024-02-19T18:58:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e88399700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e800feb00 con 0x7f0e800fcfc0 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e86135700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fbe50 0x7f0e800fff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e86135700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fbe50 0x7f0e800fff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35082/0 (socket says 172.21.15.77:35082) 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e86135700 1 -- 172.21.15.77:0/3248389793 learned_addr learned my addr 172.21.15.77:0/3248389793 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e85934700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e800fcfc0 0x7f0e800fe390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e86135700 1 -- 172.21.15.77:0/3248389793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e800fcfc0 msgr2=0x7f0e800fe390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e86135700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e800fcfc0 0x7f0e800fe390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e86135700 1 -- 172.21.15.77:0/3248389793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70005040 con 0x7f0e800fbe50 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.188+0000 7f0e85934700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e800fcfc0 0x7f0e800fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.189+0000 7f0e86135700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fbe50 0x7f0e800fff10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c002a50 tx=0x7f0e7c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.189+0000 7f0e76ffd700 1 -- 172.21.15.77:0/3248389793 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7c007cc0 con 0x7f0e800fbe50 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.189+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e800fede0 con 0x7f0e800fbe50 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.189+0000 7f0e76ffd700 1 -- 172.21.15.77:0/3248389793 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e7c00f040 con 0x7f0e800fbe50 2024-02-19T18:58:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.189+0000 7f0e76ffd700 1 -- 172.21.15.77:0/3248389793 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e7c00da30 con 0x7f0e800fbe50 2024-02-19T18:58:41.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.189+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e80115980 con 0x7f0e800fbe50 2024-02-19T18:58:41.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.190+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e8004bed0 con 0x7f0e800fbe50 2024-02-19T18:58:41.210 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.190+0000 7f0e76ffd700 1 -- 172.21.15.77:0/3248389793 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f0e7c00db90 con 0x7f0e800fbe50 2024-02-19T18:58:41.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.191+0000 7f0e76ffd700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0e6c0692b0 0x7f0e6c06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:41.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.191+0000 7f0e76ffd700 1 -- 172.21.15.77:0/3248389793 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0e7c086f40 con 0x7f0e800fbe50 2024-02-19T18:58:41.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.191+0000 7f0e85934700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0e6c0692b0 0x7f0e6c06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:41.211 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.191+0000 7f0e85934700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0e6c0692b0 0x7f0e6c06b760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b3b0 tx=0x7f0e70009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:41.213 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.195+0000 7f0e76ffd700 1 -- 172.21.15.77:0/3248389793 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f0e7c056c80 con 0x7f0e800fbe50 2024-02-19T18:58:41.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:41 smithi077 ceph-mon[29247]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-02-19T18:58:41.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:41.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:41.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:41.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:41.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:41.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:41 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:41.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.442+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e80104940 con 0x7f0e800fbe50 2024-02-19T18:58:41.460 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.442+0000 7f0e76ffd700 1 -- 172.21.15.77:0/3248389793 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4692 (secure 0 0 0) 0x7f0e7c056810 con 0x7f0e800fbe50 2024-02-19T18:58:41.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0e6c0692b0 msgr2=0x7f0e6c06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:41.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0e6c0692b0 0x7f0e6c06b760 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0e7000b3b0 tx=0x7f0e70009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:41.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fbe50 msgr2=0x7f0e800fff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:41.462 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fbe50 0x7f0e800fff10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c002a50 tx=0x7f0e7c0046b0 comp rx=0 tx=0).stop 2024-02-19T18:58:41.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 shutdown_connections 2024-02-19T18:58:41.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0e6c0692b0 0x7f0e6c06b760 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:41.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0e800fbe50 0x7f0e800fff10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:41.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 --2- 172.21.15.77:0/3248389793 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0e800fcfc0 0x7f0e800fe390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:41.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 >> 172.21.15.77:0/3248389793 conn(0x7f0e800f7510 msgr2=0x7f0e80065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:41.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 shutdown_connections 2024-02-19T18:58:41.463 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:41.444+0000 7f0e88399700 1 -- 172.21.15.77:0/3248389793 wait complete. 2024-02-19T18:58:41.463 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-02-19T18:58:41.472 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T18:58:41.482 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:41 smithi104 ceph-mon[42320]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-02-19T18:58:41.482 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:41.482 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:41.482 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:41.482 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:41.482 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:41.482 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:41 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:58:42.503 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:42 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3248389793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T18:58:42.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:42 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3248389793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T18:58:42.563 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-02-19T18:58:43.353 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:43.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:43 smithi077 ceph-mon[29247]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-02-19T18:58:43.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:43 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:43.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:43 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:43.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:43 smithi104 ceph-mon[42320]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-02-19T18:58:43.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:43 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:43.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:43 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:45.488 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:45 smithi077 ceph-mon[29247]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-02-19T18:58:45.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:45 smithi104 ceph-mon[42320]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 0 B/s wr, 0 op/s 2024-02-19T18:58:45.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.742+0000 7f8001736700 1 -- 172.21.15.77:0/1757475135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc0fb220 msgr2=0x7f7ffc0fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:45.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.742+0000 7f8001736700 1 --2- 172.21.15.77:0/1757475135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc0fb220 0x7f7ffc0fb630 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f7fec004940 tx=0x7f7fec009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:45.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.742+0000 7f8001736700 1 -- 172.21.15.77:0/1757475135 shutdown_connections 2024-02-19T18:58:45.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.742+0000 7f8001736700 1 --2- 172.21.15.77:0/1757475135 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ffc0fe2b0 0x7f7ffc0fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:45.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.742+0000 7f8001736700 1 --2- 172.21.15.77:0/1757475135 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc0fb220 0x7f7ffc0fb630 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:45.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.742+0000 7f8001736700 1 -- 172.21.15.77:0/1757475135 >> 172.21.15.77:0/1757475135 conn(0x7f7ffc0f6ca0 msgr2=0x7f7ffc0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:45.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.742+0000 7f8001736700 1 -- 172.21.15.77:0/1757475135 shutdown_connections 2024-02-19T18:58:45.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.742+0000 7f8001736700 1 -- 172.21.15.77:0/1757475135 wait complete. 2024-02-19T18:58:45.763 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f8001736700 1 Processor -- start 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f8001736700 1 -- start start 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f8001736700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ffc0fe2b0 0x7f7ffc10dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f8001736700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc10e120 0x7f7ffc10efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f8001736700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc10e670 con 0x7f7ffc10e120 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f8001736700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc10e7b0 con 0x7f7ffc0fe2b0 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f7ffa7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc10e120 0x7f7ffc10efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f7ffa7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc10e120 0x7f7ffc10efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:35090/0 (socket says 172.21.15.77:35090) 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f7ffa7fc700 1 -- 172.21.15.77:0/921259059 learned_addr learned my addr 172.21.15.77:0/921259059 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:45.764 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.744+0000 7f7ffaffd700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ffc0fe2b0 0x7f7ffc10dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f7ffa7fc700 1 -- 172.21.15.77:0/921259059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ffc0fe2b0 msgr2=0x7f7ffc10dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f7ffa7fc700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ffc0fe2b0 0x7f7ffc10dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f7ffa7fc700 1 -- 172.21.15.77:0/921259059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec005040 con 0x7f7ffc10e120 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f7ffa7fc700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc10e120 0x7f7ffc10efc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000b790 tx=0x7f7ff0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f7fe37fe700 1 -- 172.21.15.77:0/921259059 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ff000f6f0 con 0x7f7ffc10e120 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f7fe37fe700 1 -- 172.21.15.77:0/921259059 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7ff000fd20 con 0x7f7ffc10e120 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f7fe37fe700 1 -- 172.21.15.77:0/921259059 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7ff00103a0 con 0x7f7ffc10e120 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ffc10f6a0 con 0x7f7ffc10e120 2024-02-19T18:58:45.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.745+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ffc10fbe0 con 0x7f7ffc10e120 2024-02-19T18:58:45.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.746+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ffc107b00 con 0x7f7ffc10e120 2024-02-19T18:58:45.766 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.747+0000 7f7fe37fe700 1 -- 172.21.15.77:0/921259059 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7ff0011400 con 0x7f7ffc10e120 2024-02-19T18:58:45.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.747+0000 7f7fe37fe700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f7fe40692a0 0x7f7fe406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:45.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.747+0000 7f7fe37fe700 1 -- 172.21.15.77:0/921259059 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7ff0086e90 con 0x7f7ffc10e120 2024-02-19T18:58:45.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.747+0000 7f7ffaffd700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f7fe40692a0 0x7f7fe406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:45.767 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.747+0000 7f7ffaffd700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f7fe40692a0 0x7f7fe406b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7fec000c00 tx=0x7f7fec009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:45.769 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:45.752+0000 7f7fe37fe700 1 -- 172.21.15.77:0/921259059 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7ff0056cf0 con 0x7f7ffc10e120 2024-02-19T18:58:46.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.013+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f7ffc04bed0 con 0x7f7ffc10e120 2024-02-19T18:58:46.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.013+0000 7f7fe37fe700 1 -- 172.21.15.77:0/921259059 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v13) v1 ==== 78+0+99 (secure 0 0 0) 0x7f7ff0056880 con 0x7f7ffc10e120 2024-02-19T18:58:46.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f7fe40692a0 msgr2=0x7f7fe406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f7fe40692a0 0x7f7fe406b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7fec000c00 tx=0x7f7fec009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:46.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc10e120 msgr2=0x7f7ffc10efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc10e120 0x7f7ffc10efc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f7ff000b790 tx=0x7f7ff0009250 comp rx=0 tx=0).stop 2024-02-19T18:58:46.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 shutdown_connections 2024-02-19T18:58:46.034 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f7fe40692a0 0x7f7fe406b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7ffc0fe2b0 0x7f7ffc10dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 --2- 172.21.15.77:0/921259059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f7ffc10e120 0x7f7ffc10efc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 >> 172.21.15.77:0/921259059 conn(0x7f7ffc0f6ca0 msgr2=0x7f7ffc0f8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:46.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 shutdown_connections 2024-02-19T18:58:46.035 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.016+0000 7f8001736700 1 -- 172.21.15.77:0/921259059 wait complete. 2024-02-19T18:58:46.045 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T18:58:46.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:46 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/921259059' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-19T18:58:46.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:46 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/921259059' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-02-19T18:58:46.669 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-02-19T18:58:46.684 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T18:58:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/2763931071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 msgr2=0x7f1ebc103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 --2- 172.21.15.77:0/2763931071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 0x7f1ebc103b10 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00048d0 tx=0x7f1eb0009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/2763931071 shutdown_connections 2024-02-19T18:58:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 --2- 172.21.15.77:0/2763931071 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 0x7f1ebc103b10 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 --2- 172.21.15.77:0/2763931071 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ebc065180 0x7f1ebc1011f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/2763931071 >> 172.21.15.77:0/2763931071 conn(0x7f1ebc0f6820 msgr2=0x7f1ebc0f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/2763931071 shutdown_connections 2024-02-19T18:58:46.801 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/2763931071 wait complete. 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 Processor -- start 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.780+0000 7f1ec0f9d640 1 -- start start 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1ec0f9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ebc065180 0x7f1ebc10ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1ec0f9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 0x7f1ebc10f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1ec0f9d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc10f840 con 0x7f1ebc101730 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1ec0f9d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc114250 con 0x7f1ebc065180 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eb9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 0x7f1ebc10f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eb9d9b640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 0x7f1ebc10f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44788/0 (socket says 172.21.15.77:44788) 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eb9d9b640 1 -- 172.21.15.77:0/201301746 learned_addr learned my addr 172.21.15.77:0/201301746 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eba59c640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ebc065180 0x7f1ebc10ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eb9d9b640 1 -- 172.21.15.77:0/201301746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ebc065180 msgr2=0x7f1ebc10ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eb9d9b640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ebc065180 0x7f1ebc10ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eb9d9b640 1 -- 172.21.15.77:0/201301746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb0005040 con 0x7f1ebc101730 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eba59c640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ebc065180 0x7f1ebc10ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eb9d9b640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 0x7f1ebc10f1f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1eb002bda0 tx=0x7f1eb0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:46.802 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eaaffd640 1 -- 172.21.15.77:0/201301746 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1eb00094c0 con 0x7f1ebc101730 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eaaffd640 1 -- 172.21.15.77:0/201301746 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1eb0007cc0 con 0x7f1ebc101730 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc114390 con 0x7f1ebc101730 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1eaaffd640 1 -- 172.21.15.77:0/201301746 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1eb003b400 con 0x7f1ebc101730 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.781+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc114860 con 0x7f1ebc101730 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.782+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc108f30 con 0x7f1ebc101730 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.783+0000 7f1eaaffd640 1 -- 172.21.15.77:0/201301746 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1eb002f070 con 0x7f1ebc101730 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.783+0000 7f1eaaffd640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1ea4069260 0x7f1ea406b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.783+0000 7f1eaaffd640 1 -- 172.21.15.77:0/201301746 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f1eb00a9720 con 0x7f1ebc101730 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.783+0000 7f1eba59c640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1ea4069260 0x7f1ea406b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:46.803 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.783+0000 7f1eba59c640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1ea4069260 0x7f1ea406b710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc1101c0 tx=0x7f1eac00b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:46.805 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.787+0000 7f1eaaffd640 1 -- 172.21.15.77:0/201301746 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1eb00759d0 con 0x7f1ebc101730 2024-02-19T18:58:46.973 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.955+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ebc106f80 con 0x7f1ebc101730 2024-02-19T18:58:46.974 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.956+0000 7f1eaaffd640 1 -- 172.21.15.77:0/201301746 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14257 (secure 0 0 0) 0x7f1eb0078ff0 con 0x7f1ebc101730 2024-02-19T18:58:46.974 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:46.974 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":46,"fsid":"6778a95a-cf57-11ee-95bb-87774f69a715","created":"2024-02-19T18:49:39.682097+0000","modified":"2024-02-19T18:58:21.944822+0000","last_up_change":"2024-02-19T18:57:02.307357+0000","last_in_change":"2024-02-19T18:56:42.906075+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-19T18:55:48.412324+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-02-19T18:58:18.289062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-02-19T18:58:19.109429+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"fbc9e22d-2cb7-4421-b6d8-0dac7325974f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6803","nonce":806127290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6805","nonce":806127290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6809","nonce":806127290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":806127290},{"type":"v1","addr":"172.21.15.77:6807","nonce":806127290}]},"public_addr":"172.21.15.77:6803/806127290","cluster_addr":"172.21.15.77:6805/806127290","heartbeat_back_addr":"172.21.15.77:6809/806127290","heartbeat_front_addr":"172.21.15.77:6807/806127290","state":["exists","up"]},{"osd":1,"uuid":"c544f75d-7ae5-4261-b55e-f2be62afb9a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6811","nonce":3114250568}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6813","nonce":3114250568}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6817","nonce":3114250568}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":3114250568},{"type":"v1","addr":"172.21.15.77:6815","nonce":3114250568}]},"public_addr":"172.21.15.77:6811/3114250568","cluster_addr":"172.21.15.77:6813/3114250568","heartbeat_back_addr":"172.21.15.77:6817/3114250568","heartbeat_front_addr":"172.21.15.77:6815/3114250568","state":["exists","up"]},{"osd":2,"uuid":"8463e27a-7421-42a1-8b87-7185b9f61766","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6819","nonce":100949402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6821","nonce":100949402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6824","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6825","nonce":100949402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":100949402},{"type":"v1","addr":"172.21.15.77:6823","nonce":100949402}]},"public_addr":"172.21.15.77:6819/100949402","cluster_addr":"172.21.15.77:6821/100949402","heartbeat_back_addr":"172.21.15.77:6825/100949402","heartbeat_front_addr":"172.21.15.77:6823/100949402","state":["exists","up"]},{"osd":3,"uuid":"d15c9aad-95fd-4446-b3da-4f447fc3f7a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6801","nonce":3308675957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6803","nonce":3308675957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6807","nonce":3308675957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3308675957},{"type":"v1","addr":"172.21.15.104:6805","nonce":3308675957}]},"public_addr":"172.21.15.104:6801/3308675957","cluster_addr":"172.21.15.104:6803/3308675957","heartbeat_back_addr":"172.21.15.104:6807/3308675957","heartbeat_front_addr":"172.21.15.104:6805/3308675957","state":["exists","up"]},{"osd":4,"uuid":"a37971a1-f37c-4136-ae69-8a9999870db4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6809","nonce":3596569551}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6811","nonce":3596569551}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6815","nonce":3596569551}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":3596569551},{"type":"v1","addr":"172.21.15.104:6813","nonce":3596569551}]},"public_addr":"172.21.15.104:6809/3596569551","cluster_addr":"172.21.15.104:6811/3596569551","heartbeat_back_addr":"172.21.15.104:6815/3596569551","heartbeat_front_addr":"172.21.15.104:6813/3596569551","state":["exists","up"]},{"osd":5,"uuid":"ffc028a3-833b-4811-a38b-a47f1c6e15da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6817","nonce":2114623660}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6819","nonce":2114623660}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6823","nonce":2114623660}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2114623660},{"type":"v1","addr":"172.21.15.104:6821","nonce":2114623660}]},"public_addr":"172.21.15.104:6817/2114623660","cluster_addr":"172.21.15.104:6819/2114623660","heartbeat_back_addr":"172.21.15.104:6823/2114623660","heartbeat_front_addr":"172.21.15.104:6821/2114623660","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:54:46.893755+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:55:15.359473+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:55:45.118036+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:08.973267+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:56:34.107665+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-02-19T18:57:00.315125+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/2151174911":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/352143230":"2024-02-20T18:53:34.087006+0000","172.21.15.77:6800/3287530536":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/4032060278":"2024-02-20T18:53:34.087006+0000","172.21.15.77:6801/1028404574":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/2030253411":"2024-02-20T18:50:15.313914+0000","172.21.15.77:0/3240479911":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/2766229332":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/1552197174":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6800/1028404574":"2024-02-20T18:50:46.166039+0000","172.21.15.77:6801/2289192383":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6801/3287530536":"2024-02-20T18:53:34.087006+0000","172.21.15.77:0/1650327672":"2024-02-20T18:50:46.166039+0000","172.21.15.77:0/1405110191":"2024-02-20T18:50:15.313914+0000","172.21.15.77:6800/2289192383":"2024-02-20T18:50:15.313914+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-19T18:58:46.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1ea4069260 msgr2=0x7f1ea406b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1ea4069260 0x7f1ea406b710 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc1101c0 tx=0x7f1eac00b040 comp rx=0 tx=0).stop 2024-02-19T18:58:46.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 msgr2=0x7f1ebc10f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:46.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 0x7f1ebc10f1f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1eb002bda0 tx=0x7f1eb0009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 shutdown_connections 2024-02-19T18:58:46.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1ea4069260 0x7f1ea406b710 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ebc101730 0x7f1ebc10f1f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 --2- 172.21.15.77:0/201301746 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ebc065180 0x7f1ebc10ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:46.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 >> 172.21.15.77:0/201301746 conn(0x7f1ebc0f6820 msgr2=0x7f1ebc0f8c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:46.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 shutdown_connections 2024-02-19T18:58:46.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:46.958+0000 7f1ec0f9d640 1 -- 172.21.15.77:0/201301746 wait complete. 2024-02-19T18:58:46.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-19T18:55:48.412324+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-02-19T18:58:18.289062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-02-19T18:58:19.109429+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-02-19T18:58:46.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-19T18:58:47.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.072+0000 7f98ddb9d640 1 -- 172.21.15.77:0/738334854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 msgr2=0x7f98d80fb580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.092 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.072+0000 7f98ddb9d640 1 --2- 172.21.15.77:0/738334854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 0x7f98d80fb580 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f98c80048d0 tx=0x7f98c8009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.072+0000 7f98ddb9d640 1 -- 172.21.15.77:0/738334854 shutdown_connections 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.072+0000 7f98ddb9d640 1 --2- 172.21.15.77:0/738334854 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d80fc440 0x7f98d80fc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.072+0000 7f98ddb9d640 1 --2- 172.21.15.77:0/738334854 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 0x7f98d80fb580 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.072+0000 7f98ddb9d640 1 -- 172.21.15.77:0/738334854 >> 172.21.15.77:0/738334854 conn(0x7f98d80f6970 msgr2=0x7f98d80f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.072+0000 7f98ddb9d640 1 -- 172.21.15.77:0/738334854 shutdown_connections 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.072+0000 7f98ddb9d640 1 -- 172.21.15.77:0/738334854 wait complete. 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98ddb9d640 1 Processor -- start 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98ddb9d640 1 -- start start 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98ddb9d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 0x7f98d810ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98ddb9d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d80fc440 0x7f98d810bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98ddb9d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d810c5f0 con 0x7f98d80fb170 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98ddb9d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d8111040 con 0x7f98d80fc440 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98d77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 0x7f98d810ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.093 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98d77fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 0x7f98d810ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44796/0 (socket says 172.21.15.77:44796) 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98d77fe640 1 -- 172.21.15.77:0/3674045778 learned_addr learned my addr 172.21.15.77:0/3674045778 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98d6ffd640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d80fc440 0x7f98d810bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98d77fe640 1 -- 172.21.15.77:0/3674045778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d80fc440 msgr2=0x7f98d810bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98d77fe640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d80fc440 0x7f98d810bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.073+0000 7f98d77fe640 1 -- 172.21.15.77:0/3674045778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c8005040 con 0x7f98d80fb170 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.074+0000 7f98d77fe640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 0x7f98d810ba60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f98c8007da0 tx=0x7f98c8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.074+0000 7f98bffff640 1 -- 172.21.15.77:0/3674045778 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c80094c0 con 0x7f98d80fb170 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.074+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d8111180 con 0x7f98d80fb170 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.074+0000 7f98bffff640 1 -- 172.21.15.77:0/3674045778 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f98c8009620 con 0x7f98d80fb170 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.074+0000 7f98bffff640 1 -- 172.21.15.77:0/3674045778 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f98c803b620 con 0x7f98d80fb170 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.074+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d8111670 con 0x7f98d80fb170 2024-02-19T18:58:47.094 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.075+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d8103ce0 con 0x7f98d80fb170 2024-02-19T18:58:47.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.075+0000 7f98bffff640 1 -- 172.21.15.77:0/3674045778 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f98c802f070 con 0x7f98d80fb170 2024-02-19T18:58:47.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.075+0000 7f98bffff640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f98c0069210 0x7f98c006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.076+0000 7f98bffff640 1 -- 172.21.15.77:0/3674045778 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f98c80a97f0 con 0x7f98d80fb170 2024-02-19T18:58:47.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.076+0000 7f98d6ffd640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f98c0069210 0x7f98c006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.095 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.076+0000 7f98d6ffd640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f98c0069210 0x7f98c006b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f98cc007fd0 tx=0x7f98cc009e90 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:47.098 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.080+0000 7f98bffff640 1 -- 172.21.15.77:0/3674045778 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f98c8075aa0 con 0x7f98d80fb170 2024-02-19T18:58:47.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.246+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f98d8103e70 con 0x7f98d80fb170 2024-02-19T18:58:47.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.247+0000 7f98bffff640 1 -- 172.21.15.77:0/3674045778 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f98d8103e70 con 0x7f98d80fb170 2024-02-19T18:58:47.265 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 1 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.248+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f98c0069210 msgr2=0x7f98c006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.248+0000 7f98ddb9d640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f98c0069210 0x7f98c006b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f98cc007fd0 tx=0x7f98cc009e90 comp rx=0 tx=0).stop 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 msgr2=0x7f98d810ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 0x7f98d810ba60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f98c8007da0 tx=0x7f98c8009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 shutdown_connections 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f98c0069210 0x7f98c006b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f98d80fc440 0x7f98d810bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 --2- 172.21.15.77:0/3674045778 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f98d80fb170 0x7f98d810ba60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 >> 172.21.15.77:0/3674045778 conn(0x7f98d80f6970 msgr2=0x7f98d80ff670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:47.267 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 shutdown_connections 2024-02-19T18:58:47.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.249+0000 7f98ddb9d640 1 -- 172.21.15.77:0/3674045778 wait complete. 2024-02-19T18:58:47.278 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 -- 172.21.15.77:0/4237406591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 msgr2=0x7f2f8c0fef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 --2- 172.21.15.77:0/4237406591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 0x7f2f8c0fef00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c0048d0 tx=0x7f2f7c009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 -- 172.21.15.77:0/4237406591 shutdown_connections 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 --2- 172.21.15.77:0/4237406591 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f8c0ff4d0 0x7f2f8c1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 --2- 172.21.15.77:0/4237406591 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 0x7f2f8c0fef00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 -- 172.21.15.77:0/4237406591 >> 172.21.15.77:0/4237406591 conn(0x7f2f8c0f68d0 msgr2=0x7f2f8c0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 -- 172.21.15.77:0/4237406591 shutdown_connections 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 -- 172.21.15.77:0/4237406591 wait complete. 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 Processor -- start 2024-02-19T18:58:47.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.364+0000 7f2f90a6e640 1 -- start start 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f90a6e640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 0x7f2f8c0fc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f90a6e640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f8c0ff4d0 0x7f2f8c0fae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f90a6e640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c0fb340 con 0x7f2f8c065da0 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f90a6e640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8c0fb4b0 con 0x7f2f8c0ff4d0 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f8a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 0x7f2f8c0fc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f8a59c640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 0x7f2f8c0fc780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44802/0 (socket says 172.21.15.77:44802) 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f8a59c640 1 -- 172.21.15.77:0/2951130505 learned_addr learned my addr 172.21.15.77:0/2951130505 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f83fff640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f8c0ff4d0 0x7f2f8c0fae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f8a59c640 1 -- 172.21.15.77:0/2951130505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f8c0ff4d0 msgr2=0x7f2f8c0fae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f8a59c640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f8c0ff4d0 0x7f2f8c0fae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f8a59c640 1 -- 172.21.15.77:0/2951130505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f7c005040 con 0x7f2f8c065da0 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.365+0000 7f2f8a59c640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 0x7f2f8c0fc780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c02bda0 tx=0x7f2f7c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:47.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.366+0000 7f2f82ffd640 1 -- 172.21.15.77:0/2951130505 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f7c0094c0 con 0x7f2f8c065da0 2024-02-19T18:58:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.366+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f8c0fb730 con 0x7f2f8c065da0 2024-02-19T18:58:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.366+0000 7f2f82ffd640 1 -- 172.21.15.77:0/2951130505 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2f7c009620 con 0x7f2f8c065da0 2024-02-19T18:58:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.366+0000 7f2f82ffd640 1 -- 172.21.15.77:0/2951130505 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2f7c03b640 con 0x7f2f8c065da0 2024-02-19T18:58:47.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.366+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f8c0fbc50 con 0x7f2f8c065da0 2024-02-19T18:58:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.367+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8c106e00 con 0x7f2f8c065da0 2024-02-19T18:58:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.367+0000 7f2f82ffd640 1 -- 172.21.15.77:0/2951130505 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f2f7c02f070 con 0x7f2f8c065da0 2024-02-19T18:58:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.368+0000 7f2f82ffd640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f2f78069210 0x7f2f7806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.368+0000 7f2f82ffd640 1 -- 172.21.15.77:0/2951130505 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2f7c0a9790 con 0x7f2f8c065da0 2024-02-19T18:58:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.368+0000 7f2f83fff640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f2f78069210 0x7f2f7806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.368+0000 7f2f83fff640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f2f78069210 0x7f2f7806b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0fc530 tx=0x7f2f7400b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:47.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.372+0000 7f2f82ffd640 1 -- 172.21.15.77:0/2951130505 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f2f7c0759e0 con 0x7f2f8c065da0 2024-02-19T18:58:47.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:47 smithi077 ceph-mon[29247]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:47.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:47 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/201301746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:58:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:47 smithi104 ceph-mon[42320]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:47.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:47 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/201301746' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-19T18:58:47.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.554+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f2f8c104e50 con 0x7f2f8c065da0 2024-02-19T18:58:47.572 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.554+0000 7f2f82ffd640 1 -- 172.21.15.77:0/2951130505 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f2f7c079000 con 0x7f2f8c065da0 2024-02-19T18:58:47.572 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f2f78069210 msgr2=0x7f2f7806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f2f78069210 0x7f2f7806b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0fc530 tx=0x7f2f7400b040 comp rx=0 tx=0).stop 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 msgr2=0x7f2f8c0fc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 0x7f2f8c0fc780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2f7c02bda0 tx=0x7f2f7c009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 shutdown_connections 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f2f78069210 0x7f2f7806b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f2f8c0ff4d0 0x7f2f8c0fae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 --2- 172.21.15.77:0/2951130505 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f2f8c065da0 0x7f2f8c0fc780 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 >> 172.21.15.77:0/2951130505 conn(0x7f2f8c0f68d0 msgr2=0x7f2f8c0f8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 shutdown_connections 2024-02-19T18:58:47.575 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.556+0000 7f2f90a6e640 1 -- 172.21.15.77:0/2951130505 wait complete. 2024-02-19T18:58:47.586 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-02-19T18:58:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.672+0000 7f06926ae640 1 -- 172.21.15.77:0/2136384378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fa670 msgr2=0x7f068c0faa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.672+0000 7f06926ae640 1 --2- 172.21.15.77:0/2136384378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fa670 0x7f068c0faa80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f067c0048d0 tx=0x7f067c009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.672+0000 7f06926ae640 1 -- 172.21.15.77:0/2136384378 shutdown_connections 2024-02-19T18:58:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.672+0000 7f06926ae640 1 --2- 172.21.15.77:0/2136384378 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0fb050 0x7f068c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.690 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.672+0000 7f06926ae640 1 --2- 172.21.15.77:0/2136384378 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fa670 0x7f068c0faa80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.672+0000 7f06926ae640 1 -- 172.21.15.77:0/2136384378 >> 172.21.15.77:0/2136384378 conn(0x7f068c0f6090 msgr2=0x7f068c0f84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:47.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.672+0000 7f06926ae640 1 -- 172.21.15.77:0/2136384378 shutdown_connections 2024-02-19T18:58:47.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.672+0000 7f06926ae640 1 -- 172.21.15.77:0/2136384378 wait complete. 2024-02-19T18:58:47.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f06926ae640 1 Processor -- start 2024-02-19T18:58:47.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f06926ae640 1 -- start start 2024-02-19T18:58:47.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f06926ae640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0fa670 0x7f068c10cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.692 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f06926ae640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fb050 0x7f068c10d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f06926ae640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c10d7d0 con 0x7f068c0fb050 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f06926ae640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f068c1121e0 con 0x7f068c0fa670 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f068b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fb050 0x7f068c10d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f068b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fb050 0x7f068c10d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44828/0 (socket says 172.21.15.77:44828) 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f068b7fe640 1 -- 172.21.15.77:0/1583755716 learned_addr learned my addr 172.21.15.77:0/1583755716 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f068bfff640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0fa670 0x7f068c10cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f068b7fe640 1 -- 172.21.15.77:0/1583755716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0fa670 msgr2=0x7f068c10cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f068b7fe640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0fa670 0x7f068c10cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.673+0000 7f068b7fe640 1 -- 172.21.15.77:0/1583755716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c005040 con 0x7f068c0fb050 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.674+0000 7f068b7fe640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fb050 0x7f068c10d180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0680009af0 tx=0x7f0680004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.674+0000 7f0688ff9640 1 -- 172.21.15.77:0/1583755716 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0680007c30 con 0x7f068c0fb050 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.674+0000 7f0688ff9640 1 -- 172.21.15.77:0/1583755716 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f068000f040 con 0x7f068c0fb050 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.674+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f068c112380 con 0x7f068c0fb050 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.674+0000 7f0688ff9640 1 -- 172.21.15.77:0/1583755716 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0680009570 con 0x7f068c0fb050 2024-02-19T18:58:47.693 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.674+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f068c112850 con 0x7f068c0fb050 2024-02-19T18:58:47.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.675+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f068c106e40 con 0x7f068c0fb050 2024-02-19T18:58:47.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.675+0000 7f0688ff9640 1 -- 172.21.15.77:0/1583755716 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f06800097b0 con 0x7f068c0fb050 2024-02-19T18:58:47.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.676+0000 7f0688ff9640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0674069210 0x7f067406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.694 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.676+0000 7f0688ff9640 1 -- 172.21.15.77:0/1583755716 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0680087190 con 0x7f068c0fb050 2024-02-19T18:58:47.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.676+0000 7f068bfff640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0674069210 0x7f067406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.695 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.676+0000 7f068bfff640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0674069210 0x7f067406b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f067c000c00 tx=0x7f067c009da0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:47.698 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.680+0000 7f0688ff9640 1 -- 172.21.15.77:0/1583755716 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f06800533a0 con 0x7f068c0fb050 2024-02-19T18:58:47.863 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.845+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f068c104e90 con 0x7f068c0fb050 2024-02-19T18:58:47.864 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.846+0000 7f0688ff9640 1 -- 172.21.15.77:0/1583755716 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f06800569c0 con 0x7f068c0fb050 2024-02-19T18:58:47.864 INFO:teuthology.orchestra.run.smithi077.stdout:pg_num: 32 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0674069210 msgr2=0x7f067406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0674069210 0x7f067406b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f067c000c00 tx=0x7f067c009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fb050 msgr2=0x7f068c10d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fb050 0x7f068c10d180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0680009af0 tx=0x7f0680004570 comp rx=0 tx=0).stop 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 shutdown_connections 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f0674069210 0x7f067406b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f068c0fb050 0x7f068c10d180 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 --2- 172.21.15.77:0/1583755716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f068c0fa670 0x7f068c10cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.847+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 >> 172.21.15.77:0/1583755716 conn(0x7f068c0f6090 msgr2=0x7f068c100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.848+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 shutdown_connections 2024-02-19T18:58:47.866 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.848+0000 7f06926ae640 1 -- 172.21.15.77:0/1583755716 wait complete. 2024-02-19T18:58:47.877 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-19T18:58:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.956+0000 7f406299d640 1 -- 172.21.15.77:0/951355144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 msgr2=0x7f405c0fc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.956+0000 7f406299d640 1 --2- 172.21.15.77:0/951355144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 0x7f405c0fc820 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f40500048d0 tx=0x7f4050009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.956+0000 7f406299d640 1 -- 172.21.15.77:0/951355144 shutdown_connections 2024-02-19T18:58:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.956+0000 7f406299d640 1 --2- 172.21.15.77:0/951355144 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 0x7f405c0fc820 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.956+0000 7f406299d640 1 --2- 172.21.15.77:0/951355144 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405c0fb180 0x7f405c0fb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.956+0000 7f406299d640 1 -- 172.21.15.77:0/951355144 >> 172.21.15.77:0/951355144 conn(0x7f405c0f6860 msgr2=0x7f405c0f8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:47.976 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.956+0000 7f406299d640 1 -- 172.21.15.77:0/951355144 shutdown_connections 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.956+0000 7f406299d640 1 -- 172.21.15.77:0/951355144 wait complete. 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f406299d640 1 Processor -- start 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f406299d640 1 -- start start 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f406299d640 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405c0fb180 0x7f405c10ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f406299d640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 0x7f405c10bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f406299d640 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c10c5c0 con 0x7f405c0fc3b0 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f406299d640 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c111010 con 0x7f405c0fb180 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f405b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 0x7f405c10bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f405b7fe640 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 0x7f405c10bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44858/0 (socket says 172.21.15.77:44858) 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f405b7fe640 1 -- 172.21.15.77:0/700070278 learned_addr learned my addr 172.21.15.77:0/700070278 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f405bfff640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405c0fb180 0x7f405c10ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f405b7fe640 1 -- 172.21.15.77:0/700070278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405c0fb180 msgr2=0x7f405c10ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f405b7fe640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405c0fb180 0x7f405c10ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.957+0000 7f405b7fe640 1 -- 172.21.15.77:0/700070278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4050005040 con 0x7f405c0fc3b0 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.958+0000 7f405b7fe640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 0x7f405c10bf70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4050007da0 tx=0x7f4050009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:47.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.958+0000 7f4058ff9640 1 -- 172.21.15.77:0/700070278 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f40500094c0 con 0x7f405c0fc3b0 2024-02-19T18:58:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.958+0000 7f4058ff9640 1 -- 172.21.15.77:0/700070278 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4050009620 con 0x7f405c0fc3b0 2024-02-19T18:58:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.958+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c111150 con 0x7f405c0fc3b0 2024-02-19T18:58:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.958+0000 7f4058ff9640 1 -- 172.21.15.77:0/700070278 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f405003b620 con 0x7f405c0fc3b0 2024-02-19T18:58:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.958+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c111640 con 0x7f405c0fc3b0 2024-02-19T18:58:47.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.959+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c103cd0 con 0x7f405c0fc3b0 2024-02-19T18:58:47.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.959+0000 7f4058ff9640 1 -- 172.21.15.77:0/700070278 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f405002f070 con 0x7f405c0fc3b0 2024-02-19T18:58:47.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.959+0000 7f4058ff9640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f4044069210 0x7f404406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:47.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.960+0000 7f405bfff640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f4044069210 0x7f404406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:47.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.960+0000 7f4058ff9640 1 -- 172.21.15.77:0/700070278 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f40500a9760 con 0x7f405c0fc3b0 2024-02-19T18:58:47.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.960+0000 7f405bfff640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f4044069210 0x7f404406b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f405c0fc210 tx=0x7f404c00b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:47.982 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:47.964+0000 7f4058ff9640 1 -- 172.21.15.77:0/700070278 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f4050075a10 con 0x7f405c0fc3b0 2024-02-19T18:58:48.183 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.165+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f405c1117d0 con 0x7f405c0fc3b0 2024-02-19T18:58:48.184 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.166+0000 7f4058ff9640 1 -- 172.21.15.77:0/700070278 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+4692 (secure 0 0 0) 0x7f4050079030 con 0x7f405c0fc3b0 2024-02-19T18:58:48.184 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-02-19T18:58:48.184 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14528,"name":"cephfs.smithi077.jmjalp","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.77:6829/3630854907","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6828","nonce":3630854907},{"type":"v1","addr":"172.21.15.77:6829","nonce":3630854907}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24253,"name":"cephfs.smithi104.rudtoq","rank":-1,"incarnation":0,"state":"up:standby","state_seq":3,"addr":"172.21.15.104:6825/2019831719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":2019831719},{"type":"v1","addr":"172.21.15.104:6825","nonce":2019831719}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24263,"name":"cephfs.smithi104.wpcqst","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.104:6827/2946362025","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":2946362025},{"type":"v1","addr":"172.21.15.104:6827","nonce":2946362025}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-02-19T18:58:19.884992+0000","modified":"2024-02-19T18:58:33.870342+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14516},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14516":{"gid":14516,"name":"cephfs.smithi077.ovassg","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.77:6827/3246598933","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6826","nonce":3246598933},{"type":"v1","addr":"172.21.15.77:6827","nonce":3246598933}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-02-19T18:58:48.184 INFO:teuthology.orchestra.run.smithi077.stderr:dumped fsmap epoch 13 2024-02-19T18:58:48.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f4044069210 msgr2=0x7f404406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:48.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f4044069210 0x7f404406b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f405c0fc210 tx=0x7f404c00b040 comp rx=0 tx=0).stop 2024-02-19T18:58:48.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 msgr2=0x7f405c10bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:48.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 0x7f405c10bf70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4050007da0 tx=0x7f4050009da0 comp rx=0 tx=0).stop 2024-02-19T18:58:48.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 shutdown_connections 2024-02-19T18:58:48.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f4044069210 0x7f404406b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:48.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f405c0fc3b0 0x7f405c10bf70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:48.186 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 --2- 172.21.15.77:0/700070278 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f405c0fb180 0x7f405c10ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:48.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 >> 172.21.15.77:0/700070278 conn(0x7f405c0f6860 msgr2=0x7f405c0ff5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:48.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 shutdown_connections 2024-02-19T18:58:48.187 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:48.168+0000 7f406299d640 1 -- 172.21.15.77:0/700070278 wait complete. 2024-02-19T18:58:48.197 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 11, 'max_mds': 1, 'flags': 18} 2024-02-19T18:58:48.197 INFO:teuthology.run_tasks:Running task kclient... 2024-02-19T18:58:48.220 INFO:tasks.kclient:Mounting kernel clients... 2024-02-19T18:58:48.221 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-02-19T18:58:48.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:48.221 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-02-19T18:58:48.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:48.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link delete ceph-brx 2024-02-19T18:58:48.306 INFO:teuthology.orchestra.run.smithi077.stderr:Cannot find device "ceph-brx" 2024-02-19T18:58:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:58:48.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:48.307 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-02-19T18:58:48.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3674045778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T18:58:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/2951130505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T18:58:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1583755716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T18:58:48.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:48 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/700070278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T18:58:48.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:48.324 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link delete ceph-brx 2024-02-19T18:58:48.395 INFO:teuthology.orchestra.run.smithi104.stderr:Cannot find device "ceph-brx" 2024-02-19T18:58:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T18:58:48.397 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-02-19T18:58:48.397 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T18:58:48.397 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T18:58:48.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-19T18:58:48.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3674045778' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-19T18:58:48.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/2951130505' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-02-19T18:58:48.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:48.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:48.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:58:48.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:58:48.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:58:48.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1583755716' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-02-19T18:58:48.422 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:48 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/700070278' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-19T18:58:48.651 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:49.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:49 smithi077 ceph-mon[29247]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:49.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:49.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:49 smithi104 ceph-mon[42320]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:49.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:58:50.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.183+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3424742475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 msgr2=0x7fe5dc0fcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:50.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.183+0000 7fe5e2ab1700 1 --2- 172.21.15.77:0/3424742475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 0x7fe5dc0fcf60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8004940 tx=0x7fe5c8009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:50.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.183+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3424742475 shutdown_connections 2024-02-19T18:58:50.203 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.183+0000 7fe5e2ab1700 1 --2- 172.21.15.77:0/3424742475 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 0x7fe5dc0fcf60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.183+0000 7fe5e2ab1700 1 --2- 172.21.15.77:0/3424742475 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5dc102f50 0x7fe5dc103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.183+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3424742475 >> 172.21.15.77:0/3424742475 conn(0x7fe5dc0f8ae0 msgr2=0x7fe5dc0faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.183+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3424742475 shutdown_connections 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.184+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3424742475 wait complete. 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.184+0000 7fe5e2ab1700 1 Processor -- start 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.184+0000 7fe5e2ab1700 1 -- start start 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.184+0000 7fe5e2ab1700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5dc102f50 0x7fe5dc113a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.184+0000 7fe5e2ab1700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 0x7fe5dc113fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.184+0000 7fe5e2ab1700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc114620 con 0x7fe5dc1038f0 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.184+0000 7fe5e2ab1700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc10d950 con 0x7fe5dc102f50 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.184+0000 7fe5dbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 0x7fe5dc113fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5dbfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 0x7fe5dc113fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44884/0 (socket says 172.21.15.77:44884) 2024-02-19T18:58:50.204 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5dbfff700 1 -- 172.21.15.77:0/3653594056 learned_addr learned my addr 172.21.15.77:0/3653594056 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5e084d700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5dc102f50 0x7fe5dc113a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5dbfff700 1 -- 172.21.15.77:0/3653594056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5dc102f50 msgr2=0x7fe5dc113a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5dbfff700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5dc102f50 0x7fe5dc113a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5dbfff700 1 -- 172.21.15.77:0/3653594056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c8005040 con 0x7fe5dc1038f0 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5dbfff700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 0x7fe5dc113fd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009150 tx=0x7fe5c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5d97fa700 1 -- 172.21.15.77:0/3653594056 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5c800e040 con 0x7fe5dc1038f0 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5dc10dbd0 con 0x7fe5dc1038f0 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5d97fa700 1 -- 172.21.15.77:0/3653594056 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5c800bbf0 con 0x7fe5dc1038f0 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5d97fa700 1 -- 172.21.15.77:0/3653594056 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5c801c3f0 con 0x7fe5dc1038f0 2024-02-19T18:58:50.205 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.185+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5dc10e110 con 0x7fe5dc1038f0 2024-02-19T18:58:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.186+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5dc105420 con 0x7fe5dc1038f0 2024-02-19T18:58:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.187+0000 7fe5d97fa700 1 -- 172.21.15.77:0/3653594056 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe5c8013420 con 0x7fe5dc1038f0 2024-02-19T18:58:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.187+0000 7fe5d97fa700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5cc0692a0 0x7fe5cc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.187+0000 7fe5d97fa700 1 -- 172.21.15.77:0/3653594056 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe5c8088560 con 0x7fe5dc1038f0 2024-02-19T18:58:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.187+0000 7fe5e084d700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5cc0692a0 0x7fe5cc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:50.206 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.187+0000 7fe5e084d700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5cc0692a0 0x7fe5cc06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0004f40 tx=0x7fe5d000b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:50.209 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.191+0000 7fe5d97fa700 1 -- 172.21.15.77:0/3653594056 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe5c8058320 con 0x7fe5dc1038f0 2024-02-19T18:58:50.435 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.417+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe5dc10f070 con 0x7fe5dc1038f0 2024-02-19T18:58:50.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.418+0000 7fe5d97fa700 1 -- 172.21.15.77:0/3653594056 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+83 (secure 0 0 0) 0x7fe5c8057eb0 con 0x7fe5dc1038f0 2024-02-19T18:58:50.436 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.419+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5cc0692a0 msgr2=0x7fe5cc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.419+0000 7fe5e2ab1700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5cc0692a0 0x7fe5cc06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe5d0004f40 tx=0x7fe5d000b040 comp rx=0 tx=0).stop 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.419+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 msgr2=0x7fe5dc113fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.419+0000 7fe5e2ab1700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 0x7fe5dc113fd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009150 tx=0x7fe5c8009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.420+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 shutdown_connections 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.420+0000 7fe5e2ab1700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe5cc0692a0 0x7fe5cc06b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.420+0000 7fe5e2ab1700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe5dc102f50 0x7fe5dc113a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.420+0000 7fe5e2ab1700 1 --2- 172.21.15.77:0/3653594056 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe5dc1038f0 0x7fe5dc113fd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.420+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 >> 172.21.15.77:0/3653594056 conn(0x7fe5dc0f8ae0 msgr2=0x7fe5dc0fa5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.420+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 shutdown_connections 2024-02-19T18:58:50.438 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:50.420+0000 7fe5e2ab1700 1 -- 172.21.15.77:0/3653594056 wait complete. 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi077.front.sepia.ceph.com 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T18:58:51.065 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-19T18:58:51.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.066 DEBUG:teuthology.orchestra.run.smithi077:> ip addr 2024-02-19T18:58:51.121 INFO:teuthology.orchestra.run.smithi077.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T18:58:51.121 INFO:teuthology.orchestra.run.smithi077.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 ::1/128 scope host 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d8 brd ff:ff:ff:ff:ff:ff 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f0 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:d8:d9:d9 brd ff:ff:ff:ff:ff:ff 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: altname enp7s0f1 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d2 brd ff:ff:ff:ff:ff:ff 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: link/ether 0c:c4:7a:8f:d3:d3 brd ff:ff:ff:ff:ff:ff 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: inet 172.21.15.77/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-19T18:58:51.122 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft 42045sec preferred_lft 42045sec 2024-02-19T18:58:51.123 INFO:teuthology.orchestra.run.smithi077.stdout: inet6 fe80::ec4:7aff:fe8f:d3d3/64 scope link 2024-02-19T18:58:51.123 INFO:teuthology.orchestra.run.smithi077.stdout: valid_lft forever preferred_lft forever 2024-02-19T18:58:51.123 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T18:58:51.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.123 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-02-19T18:58:51.123 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-02-19T18:58:51.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add name ceph-brx type bridge 2024-02-19T18:58:51.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr flush dev ceph-brx 2024-02-19T18:58:51.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set ceph-brx up 2024-02-19T18:58:51.123 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T18:58:51.124 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-02-19T18:58:51.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:51 smithi077 ceph-mon[29247]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:51.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:51 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3653594056' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T18:58:51.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.292 DEBUG:teuthology.orchestra.run.smithi077:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T18:58:51.319 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-02-19T18:58:51.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.321 DEBUG:teuthology.orchestra.run.smithi077:> route 2024-02-19T18:58:51.392 INFO:teuthology.orchestra.run.smithi077.stdout:Kernel IP routing table 2024-02-19T18:58:51.435 INFO:teuthology.orchestra.run.smithi077.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T18:58:51.435 INFO:teuthology.orchestra.run.smithi077.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T18:58:51.435 INFO:teuthology.orchestra.run.smithi077.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T18:58:51.435 INFO:teuthology.orchestra.run.smithi077.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T18:58:51.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.436 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-02-19T18:58:51.436 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-02-19T18:58:51.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T18:58:51.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T18:58:51.436 DEBUG:teuthology.orchestra.run.smithi077:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T18:58:51.436 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-02-19T18:58:51.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:51 smithi104 ceph-mon[42320]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:51.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:51 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3653594056' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T18:58:51.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.657 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list 2024-02-19T18:58:51.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.671 DEBUG:teuthology.orchestra.run.smithi077:> ip netns list-id 2024-02-19T18:58:51.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.726 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-02-19T18:58:51.726 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-02-19T18:58:51.726 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-19T18:58:51.726 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-19T18:58:51.726 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-02-19T18:58:51.808 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-19T18:58:51.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.808 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-02-19T18:58:51.808 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-02-19T18:58:51.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-19T18:58:51.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T18:58:51.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-19T18:58:51.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-19T18:58:51.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-19T18:58:51.808 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-02-19T18:58:51.949 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:51.949 DEBUG:teuthology.orchestra.run.smithi077:> (cd / && exec bash -c ' 2024-02-19T18:58:51.949 DEBUG:teuthology.orchestra.run.smithi077:> set -e 2024-02-19T18:58:51.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set brx.0 up 2024-02-19T18:58:51.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T18:58:51.949 DEBUG:teuthology.orchestra.run.smithi077:> ') 2024-02-19T18:58:51.990 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T18:58:51.990 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-19T18:58:52.045 INFO:teuthology.orchestra.run.smithi077.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-19T18:58:52.046 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T18:58:52.046 DEBUG:teuthology.orchestra.run.smithi077:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-19T18:58:52.100 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T18:58:52.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:52.100 DEBUG:teuthology.orchestra.run.smithi077:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T18:58:52.189 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T18:58:52.189 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T18:58:52.189 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-02-19T18:58:52.306 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T18:58:52.306 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T18:58:52.306 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-02-19T18:58:52.306 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.104:3300" 2024-02-19T18:58:52.306 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T18:58:52.310 INFO:teuthology.orchestra.run.smithi077.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-02-19T18:58:52.310 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout:invalid new device string format 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.104:3300" 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.104:6789" 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.104:6789:/ 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=6778a95a-cf57-11ee-95bb-87774f69a715" will pass to kernel 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout: You just mounted a file system that supports labels which does not 2024-02-19T18:58:52.311 INFO:teuthology.orchestra.run.smithi077.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-19T18:58:52.312 INFO:teuthology.orchestra.run.smithi077.stdout: applications will generate AVC messages and not be allowed access to 2024-02-19T18:58:52.312 INFO:teuthology.orchestra.run.smithi077.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-19T18:58:52.313 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T18:58:52.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:52.314 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-19T18:58:52.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:52.341 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T18:58:52.400 INFO:teuthology.orchestra.run.smithi077.stdout:client14578 2024-02-19T18:58:52.401 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T18:58:52.401 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T18:58:52.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:52.457 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T18:58:52.516 INFO:teuthology.orchestra.run.smithi077.stdout:client14578 2024-02-19T18:58:52.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:52.517 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T18:58:52.577 INFO:teuthology.orchestra.run.smithi077.stdout:client14578 2024-02-19T18:58:52.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:52.578 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/6778a95a-cf57-11ee-95bb-87774f69a715.client14578/status') 2024-02-19T18:58:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:instance: client.14578 (3)192.168.0.1:0/1583241153 2024-02-19T18:58:52.647 INFO:teuthology.orchestra.run.smithi077.stdout:blocklisted: false 2024-02-19T18:58:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records in 2024-02-19T18:58:52.647 INFO:teuthology.orchestra.run.smithi077.stderr:0+1 records out 2024-02-19T18:58:52.648 INFO:teuthology.orchestra.run.smithi077.stderr:70 bytes copied, 2.2565e-05 s, 3.1 MB/s 2024-02-19T18:58:52.649 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-02-19T18:58:52.649 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-19T18:58:52.649 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T18:58:52.649 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -- ceph fs ls 2024-02-19T18:58:52.945 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:53.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:53 smithi077 ceph-mon[29247]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:53.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:53 smithi104 ceph-mon[42320]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.582+0000 7f756c97f700 1 -- 172.21.15.77:0/162389341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 msgr2=0x7f7568070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:54.632 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.582+0000 7f756c97f700 1 --2- 172.21.15.77:0/162389341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 0x7f7568070680 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7558009ce0 tx=0x7f7558004d10 comp rx=0 tx=0).stop 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.582+0000 7f756c97f700 1 -- 172.21.15.77:0/162389341 shutdown_connections 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.582+0000 7f756c97f700 1 --2- 172.21.15.77:0/162389341 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7568070bc0 0x7f75681092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.582+0000 7f756c97f700 1 --2- 172.21.15.77:0/162389341 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 0x7f7568070680 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.582+0000 7f756c97f700 1 -- 172.21.15.77:0/162389341 >> 172.21.15.77:0/162389341 conn(0x7f75680f8ae0 msgr2=0x7f75680faef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.582+0000 7f756c97f700 1 -- 172.21.15.77:0/162389341 shutdown_connections 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.582+0000 7f756c97f700 1 -- 172.21.15.77:0/162389341 wait complete. 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756c97f700 1 Processor -- start 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756c97f700 1 -- start start 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756c97f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 0x7f7568115e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756c97f700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7568070bc0 0x7f7568116370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756c97f700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75681169c0 con 0x7f75680702b0 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756c97f700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f756810fcd0 con 0x7f7568070bc0 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 0x7f7568115e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:54.633 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756659c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 0x7f7568115e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:44902/0 (socket says 172.21.15.77:44902) 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756659c700 1 -- 172.21.15.77:0/844281660 learned_addr learned my addr 172.21.15.77:0/844281660 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f7565d9b700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7568070bc0 0x7f7568116370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756659c700 1 -- 172.21.15.77:0/844281660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7568070bc0 msgr2=0x7f7568116370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756659c700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7568070bc0 0x7f7568116370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756659c700 1 -- 172.21.15.77:0/844281660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558009940 con 0x7f75680702b0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f7565d9b700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7568070bc0 0x7f7568116370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.583+0000 7f756659c700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 0x7f7568115e30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f755800be90 tx=0x7f7558004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.584+0000 7f755effd700 1 -- 172.21.15.77:0/844281660 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755800e040 con 0x7f75680702b0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.584+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f756810ff50 con 0x7f75680702b0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.584+0000 7f755effd700 1 -- 172.21.15.77:0/844281660 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7558008ea0 con 0x7f75680702b0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.584+0000 7f755effd700 1 -- 172.21.15.77:0/844281660 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f755801c7a0 con 0x7f75680702b0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.584+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7568110490 con 0x7f75680702b0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.585+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75681069a0 con 0x7f75680702b0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.585+0000 7f755effd700 1 -- 172.21.15.77:0/844281660 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f7558012410 con 0x7f75680702b0 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.585+0000 7f755effd700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f75540692a0 0x7f755406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:54.634 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.585+0000 7f755effd700 1 -- 172.21.15.77:0/844281660 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7558088df0 con 0x7f75680702b0 2024-02-19T18:58:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.586+0000 7f7565d9b700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f75540692a0 0x7f755406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.586+0000 7f7565d9b700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f75540692a0 0x7f755406b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7550009fd0 tx=0x7f755000b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:54.635 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.590+0000 7f755effd700 1 -- 172.21.15.77:0/844281660 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f7558058b30 con 0x7f75680702b0 2024-02-19T18:58:54.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.819+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f756804bed0 con 0x7f75680702b0 2024-02-19T18:58:54.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.820+0000 7f755effd700 1 -- 172.21.15.77:0/844281660 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v13) v1 ==== 53+0+83 (secure 0 0 0) 0x7f75580586c0 con 0x7f75680702b0 2024-02-19T18:58:54.838 INFO:teuthology.orchestra.run.smithi077.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-02-19T18:58:54.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f75540692a0 msgr2=0x7f755406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:54.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f75540692a0 0x7f755406b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7550009fd0 tx=0x7f755000b040 comp rx=0 tx=0).stop 2024-02-19T18:58:54.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 msgr2=0x7f7568115e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:54.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 0x7f7568115e30 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f755800be90 tx=0x7f7558004d10 comp rx=0 tx=0).stop 2024-02-19T18:58:54.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 shutdown_connections 2024-02-19T18:58:54.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f75540692a0 0x7f755406b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f75680702b0 0x7f7568115e30 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 --2- 172.21.15.77:0/844281660 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f7568070bc0 0x7f7568116370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:54.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 >> 172.21.15.77:0/844281660 conn(0x7f75680f8ae0 msgr2=0x7f7568103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:54.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 shutdown_connections 2024-02-19T18:58:54.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:54.822+0000 7f756c97f700 1 -- 172.21.15.77:0/844281660 wait complete. 2024-02-19T18:58:55.431 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:55 smithi077 ceph-mon[29247]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:55.431 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:55 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/844281660' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi104.front.sepia.ceph.com 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-19T18:58:55.457 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-02-19T18:58:55.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:55.458 DEBUG:teuthology.orchestra.run.smithi104:> ip addr 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout: inet 127.0.0.1/8 scope host lo 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 ::1/128 scope host 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1c brd ff:ff:ff:ff:ff:ff 2024-02-19T18:58:55.478 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f0 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:d8:a1:1d brd ff:ff:ff:ff:ff:ff 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout: altname enp7s0f1 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:88 brd ff:ff:ff:ff:ff:ff 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout: link/ether 0c:c4:7a:8f:ce:89 brd ff:ff:ff:ff:ff:ff 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout: inet 172.21.15.104/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft 42028sec preferred_lft 42028sec 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout: inet6 fe80::ec4:7aff:fe8f:ce89/64 scope link 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run.smithi104.stdout: valid_lft forever preferred_lft forever 2024-02-19T18:58:55.479 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-19T18:58:55.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:55.480 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2024-02-19T18:58:55.480 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-02-19T18:58:55.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add name ceph-brx type bridge 2024-02-19T18:58:55.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr flush dev ceph-brx 2024-02-19T18:58:55.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set ceph-brx up 2024-02-19T18:58:55.480 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-19T18:58:55.480 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-02-19T18:58:55.546 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:55 smithi104 ceph-mon[42320]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:55.546 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:55 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/844281660' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-19T18:58:55.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:55.652 DEBUG:teuthology.orchestra.run.smithi104:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-19T18:58:55.720 INFO:teuthology.orchestra.run.smithi104.stdout:1 2024-02-19T18:58:55.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:55.722 DEBUG:teuthology.orchestra.run.smithi104:> route 2024-02-19T18:58:55.792 INFO:teuthology.orchestra.run.smithi104.stdout:Kernel IP routing table 2024-02-19T18:58:55.792 INFO:teuthology.orchestra.run.smithi104.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-19T18:58:55.792 INFO:teuthology.orchestra.run.smithi104.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-02-19T18:58:55.792 INFO:teuthology.orchestra.run.smithi104.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-02-19T18:58:55.792 INFO:teuthology.orchestra.run.smithi104.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-19T18:58:55.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:55.793 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2024-02-19T18:58:55.793 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-02-19T18:58:55.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-02-19T18:58:55.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-02-19T18:58:55.793 DEBUG:teuthology.orchestra.run.smithi104:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-02-19T18:58:55.793 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-02-19T18:58:56.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.074 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list 2024-02-19T18:58:56.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.089 DEBUG:teuthology.orchestra.run.smithi104:> ip netns list-id 2024-02-19T18:58:56.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.144 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2024-02-19T18:58:56.144 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-02-19T18:58:56.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-02-19T18:58:56.144 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-02-19T18:58:56.144 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-02-19T18:58:56.230 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-02-19T18:58:56.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.230 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2024-02-19T18:58:56.230 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-02-19T18:58:56.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-02-19T18:58:56.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-19T18:58:56.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-02-19T18:58:56.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-02-19T18:58:56.230 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-02-19T18:58:56.230 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-02-19T18:58:56.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.370 DEBUG:teuthology.orchestra.run.smithi104:> (cd / && exec bash -c ' 2024-02-19T18:58:56.370 DEBUG:teuthology.orchestra.run.smithi104:> set -e 2024-02-19T18:58:56.370 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set brx.0 up 2024-02-19T18:58:56.370 DEBUG:teuthology.orchestra.run.smithi104:> sudo ip link set dev brx.0 master ceph-brx 2024-02-19T18:58:56.370 DEBUG:teuthology.orchestra.run.smithi104:> ') 2024-02-19T18:58:56.410 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T18:58:56.410 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-02-19T18:58:56.465 INFO:teuthology.orchestra.run.smithi104.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-02-19T18:58:56.465 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-19T18:58:56.465 DEBUG:teuthology.orchestra.run.smithi104:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-02-19T18:58:56.519 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-02-19T18:58:56.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.520 DEBUG:teuthology.orchestra.run.smithi104:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T18:58:56.612 INFO:teuthology.orchestra.run.smithi104.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T18:58:56.612 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T18:58:56.612 INFO:teuthology.orchestra.run.smithi104.stdout:invalid new device string format 2024-02-19T18:58:56.729 INFO:teuthology.orchestra.run.smithi104.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T18:58:56.730 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T18:58:56.730 INFO:teuthology.orchestra.run.smithi104.stdout:invalid new device string format 2024-02-19T18:58:56.730 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.104:3300" 2024-02-19T18:58:56.730 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T18:58:56.734 INFO:teuthology.orchestra.run.smithi104.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-02-19T18:58:56.734 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-02-19T18:58:56.734 INFO:teuthology.orchestra.run.smithi104.stdout:invalid new device string format 2024-02-19T18:58:56.734 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: resolved to: "172.21.15.77:3300,172.21.15.104:3300" 2024-02-19T18:58:56.734 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: switching to using v1 address with old syntax 2024-02-19T18:58:56.735 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: resolved to: "172.21.15.77:6789,172.21.15.104:6789" 2024-02-19T18:58:56.735 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.77:6789,172.21.15.104:6789:/ 2024-02-19T18:58:56.735 INFO:teuthology.orchestra.run.smithi104.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=6778a95a-cf57-11ee-95bb-87774f69a715" will pass to kernel 2024-02-19T18:58:56.736 INFO:teuthology.orchestra.run.smithi104.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-02-19T18:58:56.736 INFO:teuthology.orchestra.run.smithi104.stdout: You just mounted a file system that supports labels which does not 2024-02-19T18:58:56.736 INFO:teuthology.orchestra.run.smithi104.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-19T18:58:56.736 INFO:teuthology.orchestra.run.smithi104.stdout: applications will generate AVC messages and not be allowed access to 2024-02-19T18:58:56.736 INFO:teuthology.orchestra.run.smithi104.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-19T18:58:56.737 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-19T18:58:56.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.737 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-02-19T18:58:56.764 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.764 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T18:58:56.821 INFO:teuthology.orchestra.run.smithi104.stdout:client14584 2024-02-19T18:58:56.821 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T18:58:56.821 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T18:58:56.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.877 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T18:58:56.933 INFO:teuthology.orchestra.run.smithi104.stdout:client14584 2024-02-19T18:58:56.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.933 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-19T18:58:56.988 INFO:teuthology.orchestra.run.smithi104.stdout:client14584 2024-02-19T18:58:56.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T18:58:56.989 DEBUG:teuthology.orchestra.run.smithi104:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/6778a95a-cf57-11ee-95bb-87774f69a715.client14584/status') 2024-02-19T18:58:57.055 INFO:teuthology.orchestra.run.smithi104.stdout:instance: client.14584 (3)172.21.15.104:0/1011532428 2024-02-19T18:58:57.055 INFO:teuthology.orchestra.run.smithi104.stdout:blocklisted: false 2024-02-19T18:58:57.055 INFO:teuthology.orchestra.run.smithi104.stderr:0+1 records in 2024-02-19T18:58:57.056 INFO:teuthology.orchestra.run.smithi104.stderr:0+1 records out 2024-02-19T18:58:57.056 INFO:teuthology.orchestra.run.smithi104.stderr:72 bytes copied, 1.9523e-05 s, 3.7 MB/s 2024-02-19T18:58:57.057 INFO:teuthology.run_tasks:Running task print... 2024-02-19T18:58:57.069 INFO:teuthology.task.print:**** done client 2024-02-19T18:58:57.069 INFO:teuthology.run_tasks:Running task parallel... 2024-02-19T18:58:57.092 INFO:teuthology.task.parallel:starting parallel... 2024-02-19T18:58:57.092 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-19T18:58:57.093 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-02-19T18:58:57.093 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T18:58:57.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-02-19T18:58:57.094 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-02-19T18:58:57.094 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-02-19T18:58:57.097 INFO:tasks.workunit:Pulling workunits from ref 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T18:58:57.098 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-19T18:58:57.098 INFO:tasks.workunit:timeout=3h 2024-02-19T18:58:57.098 INFO:tasks.workunit:cleanup=True 2024-02-19T18:58:57.098 DEBUG:teuthology.orchestra.run.smithi077:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-02-19 18:58:23.964396157 +0000 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-02-19 18:58:52.321580609 +0000 2024-02-19T18:58:57.118 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: 2024-02-19 18:58:23.964396157 +0000 2024-02-19T18:58:57.118 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-19T18:58:57.119 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-19T18:58:57.191 DEBUG:teuthology.orchestra.run.smithi104:> stat -- /home/ubuntu/cephtest/mnt.1 2024-02-19T18:58:57.206 INFO:teuthology.orchestra.run.smithi104.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-02-19T18:58:57.207 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-02-19T18:58:57.207 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-02-19T18:58:57.207 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-19T18:58:57.207 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-19T18:58:57.207 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-02-19T18:58:57.207 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-02-19 18:58:57.168471350 +0000 2024-02-19T18:58:57.207 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-02-19 18:58:57.168471350 +0000 2024-02-19T18:58:57.207 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-02-19 18:58:23.964396157 +0000 2024-02-19T18:58:57.207 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-02-19T18:58:57.207 DEBUG:teuthology.orchestra.run.smithi104:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-02-19T18:58:57.277 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T18:58:57.278 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://github.com/batrick/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout 1ce7ea8877c709fa862eac608550374bd96e166e 2024-02-19T18:58:57.293 INFO:tasks.workunit.client.0.smithi077.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-19T18:58:57.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:57 smithi104 ceph-mon[42320]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:57.327 INFO:tasks.workunit.client.1.smithi104.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-02-19T18:58:57.376 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:58:57.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:57 smithi077 ceph-mon[29247]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:58:59.022 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.002+0000 7f48f51b0700 1 -- 172.21.15.77:0/1214562298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 msgr2=0x7f48f00fe4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.002+0000 7f48f51b0700 1 --2- 172.21.15.77:0/1214562298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 0x7f48f00fe4f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f48e4004940 tx=0x7f48e4009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.002+0000 7f48f51b0700 1 -- 172.21.15.77:0/1214562298 shutdown_connections 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.002+0000 7f48f51b0700 1 --2- 172.21.15.77:0/1214562298 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 0x7f48f00fe4f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.002+0000 7f48f51b0700 1 --2- 172.21.15.77:0/1214562298 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48f00fb340 0x7f48f00fb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.002+0000 7f48f51b0700 1 -- 172.21.15.77:0/1214562298 >> 172.21.15.77:0/1214562298 conn(0x7f48f00f6d20 msgr2=0x7f48f00f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.002+0000 7f48f51b0700 1 -- 172.21.15.77:0/1214562298 shutdown_connections 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.002+0000 7f48f51b0700 1 -- 172.21.15.77:0/1214562298 wait complete. 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48f51b0700 1 Processor -- start 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48f51b0700 1 -- start start 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48f51b0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48f00fb340 0x7f48f010dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:59.023 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48f51b0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 0x7f48f010e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48f51b0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f010e9e0 con 0x7f48f00fbc90 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48f51b0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f010f280 con 0x7f48f00fb340 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48ee59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 0x7f48f010e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48ee59c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 0x7f48f010e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60600/0 (socket says 172.21.15.77:60600) 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48ee59c700 1 -- 172.21.15.77:0/2136787716 learned_addr learned my addr 172.21.15.77:0/2136787716 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.003+0000 7f48eed9d700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48f00fb340 0x7f48f010dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48ee59c700 1 -- 172.21.15.77:0/2136787716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48f00fb340 msgr2=0x7f48f010dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48ee59c700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48f00fb340 0x7f48f010dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48ee59c700 1 -- 172.21.15.77:0/2136787716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48e4005040 con 0x7f48f00fbc90 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48eed9d700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48f00fb340 0x7f48f010dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48ee59c700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 0x7f48f010e3e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f48e40093b0 tx=0x7f48e4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48d77fe700 1 -- 172.21.15.77:0/2136787716 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48e400e040 con 0x7f48f00fbc90 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f010f500 con 0x7f48f00fbc90 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48d77fe700 1 -- 172.21.15.77:0/2136787716 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f48e400bcc0 con 0x7f48f00fbc90 2024-02-19T18:58:59.024 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48d77fe700 1 -- 172.21.15.77:0/2136787716 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f48e401c630 con 0x7f48f00fbc90 2024-02-19T18:58:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.004+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f010fa40 con 0x7f48f00fbc90 2024-02-19T18:58:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.005+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f0107cb0 con 0x7f48f00fbc90 2024-02-19T18:58:59.025 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.005+0000 7f48d77fe700 1 -- 172.21.15.77:0/2136787716 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f48e40139f0 con 0x7f48f00fbc90 2024-02-19T18:58:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.006+0000 7f48d77fe700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f48d80692a0 0x7f48d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:58:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.006+0000 7f48d77fe700 1 -- 172.21.15.77:0/2136787716 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f48e4088590 con 0x7f48f00fbc90 2024-02-19T18:58:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.006+0000 7f48eed9d700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f48d80692a0 0x7f48d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:58:59.026 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.006+0000 7f48eed9d700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f48d80692a0 0x7f48d806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009fd0 tx=0x7f48e0009f20 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:58:59.028 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.010+0000 7f48d77fe700 1 -- 172.21.15.77:0/2136787716 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f48e4058350 con 0x7f48f00fbc90 2024-02-19T18:58:59.058 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:58:58 smithi077 ceph-mon[29247]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-19T18:58:59.244 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.226+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7f48f004bed0 con 0x7f48f00fbc90 2024-02-19T18:58:59.245 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.227+0000 7f48d77fe700 1 -- 172.21.15.77:0/2136787716 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7f48e4057ee0 con 0x7f48f00fbc90 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.228+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f48d80692a0 msgr2=0x7f48d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.228+0000 7f48f51b0700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f48d80692a0 0x7f48d806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f48e0009fd0 tx=0x7f48e0009f20 comp rx=0 tx=0).stop 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.228+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 msgr2=0x7f48f010e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.228+0000 7f48f51b0700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 0x7f48f010e3e0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f48e40093b0 tx=0x7f48e4009e30 comp rx=0 tx=0).stop 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.229+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 shutdown_connections 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.229+0000 7f48f51b0700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f48d80692a0 0x7f48d806b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.229+0000 7f48f51b0700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f48f00fb340 0x7f48f010dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.229+0000 7f48f51b0700 1 --2- 172.21.15.77:0/2136787716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f48f00fbc90 0x7f48f010e3e0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:58:59.247 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.229+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 >> 172.21.15.77:0/2136787716 conn(0x7f48f00f6d20 msgr2=0x7f48f01018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:58:59.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.229+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 shutdown_connections 2024-02-19T18:58:59.248 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:58:59.229+0000 7f48f51b0700 1 -- 172.21.15.77:0/2136787716 wait complete. 2024-02-19T18:58:59.312 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:58:58 smithi104 ceph-mon[42320]: pgmap v196: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-19T18:58:59.939 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-02-19T18:59:00.237 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:59:01.514 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:01 smithi077 ceph-mon[29247]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-19T18:59:01.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:01 smithi104 ceph-mon[42320]: pgmap v197: 65 pgs: 65 active+clean; 580 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 341 B/s wr, 0 op/s 2024-02-19T18:59:02.117 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.094+0000 7f1ce49d4700 1 -- 172.21.15.77:0/4191251178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 msgr2=0x7f1ce0102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.094+0000 7f1ce49d4700 1 --2- 172.21.15.77:0/4191251178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 0x7f1ce0102520 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4004940 tx=0x7f1cd4009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.094+0000 7f1ce49d4700 1 -- 172.21.15.77:0/4191251178 shutdown_connections 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.094+0000 7f1ce49d4700 1 --2- 172.21.15.77:0/4191251178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 0x7f1ce0102520 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.094+0000 7f1ce49d4700 1 --2- 172.21.15.77:0/4191251178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce00663e0 0x7f1ce00ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.094+0000 7f1ce49d4700 1 -- 172.21.15.77:0/4191251178 >> 172.21.15.77:0/4191251178 conn(0x7f1ce00f74e0 msgr2=0x7f1ce00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.094+0000 7f1ce49d4700 1 -- 172.21.15.77:0/4191251178 shutdown_connections 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1ce49d4700 1 -- 172.21.15.77:0/4191251178 wait complete. 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1ce49d4700 1 Processor -- start 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1ce49d4700 1 -- start start 2024-02-19T18:59:02.118 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1ce49d4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce00663e0 0x7f1ce00fd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1ce49d4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 0x7f1ce00fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1ce49d4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce00fbff0 con 0x7f1ce01000c0 2024-02-19T18:59:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1ce49d4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce00fc160 con 0x7f1ce00663e0 2024-02-19T18:59:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1cddd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 0x7f1ce00fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1cddd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 0x7f1ce00fb9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60618/0 (socket says 172.21.15.77:60618) 2024-02-19T18:59:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.095+0000 7f1cddd9b700 1 -- 172.21.15.77:0/3716394988 learned_addr learned my addr 172.21.15.77:0/3716394988 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:59:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1cde59c700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce00663e0 0x7f1ce00fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1cddd9b700 1 -- 172.21.15.77:0/3716394988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce00663e0 msgr2=0x7f1ce00fd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1cddd9b700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce00663e0 0x7f1ce00fd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1cddd9b700 1 -- 172.21.15.77:0/3716394988 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd4005040 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1cddd9b700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 0x7f1ce00fb9f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400b3b0 tx=0x7f1cd4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1cceffd700 1 -- 172.21.15.77:0/3716394988 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cd400e040 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1cceffd700 1 -- 172.21.15.77:0/3716394988 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1cd400bcc0 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1cceffd700 1 -- 172.21.15.77:0/3716394988 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1cd401c630 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce00fc3e0 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.096+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce01159f0 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.097+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce0107a70 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.098+0000 7f1cceffd700 1 -- 172.21.15.77:0/3716394988 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f1cd4013980 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.098+0000 7f1cceffd700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1cc80692a0 0x7f1cc806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.098+0000 7f1cde59c700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1cc80692a0 0x7f1cc806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.098+0000 7f1cceffd700 1 -- 172.21.15.77:0/3716394988 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f1cd4088540 con 0x7f1ce01000c0 2024-02-19T18:59:02.120 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.098+0000 7f1cde59c700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1cc80692a0 0x7f1cc806b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00fd300 tx=0x7f1cd000b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:02.121 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.102+0000 7f1cceffd700 1 -- 172.21.15.77:0/3716394988 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f1cd4058450 con 0x7f1ce01000c0 2024-02-19T18:59:02.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.294+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f1ce004bed0 con 0x7f1ce01000c0 2024-02-19T18:59:02.312 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.294+0000 7f1cceffd700 1 -- 172.21.15.77:0/3716394988 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f1cd4057fe0 con 0x7f1ce01000c0 2024-02-19T18:59:02.314 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1cc80692a0 msgr2=0x7f1cc806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1cc80692a0 0x7f1cc806b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00fd300 tx=0x7f1cd000b040 comp rx=0 tx=0).stop 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 msgr2=0x7f1ce00fb9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 0x7f1ce00fb9f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f1cd400b3b0 tx=0x7f1cd4009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 shutdown_connections 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f1cc80692a0 0x7f1cc806b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1ce00663e0 0x7f1ce00fd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 --2- 172.21.15.77:0/3716394988 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1ce01000c0 0x7f1ce00fb9f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 >> 172.21.15.77:0/3716394988 conn(0x7f1ce00f74e0 msgr2=0x7f1ce00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 shutdown_connections 2024-02-19T18:59:02.315 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:02.296+0000 7f1ce49d4700 1 -- 172.21.15.77:0/3716394988 wait complete. 2024-02-19T18:59:03.008 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph config set global log_to_journald false --force' 2024-02-19T18:59:03.267 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:59:03.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:03 smithi077 ceph-mon[29247]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-19T18:59:03.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:03 smithi104 ceph-mon[42320]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 682 B/s wr, 0 op/s 2024-02-19T18:59:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:04.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:05.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.017+0000 7f3116151700 1 -- 172.21.15.77:0/1134489128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 msgr2=0x7f3110102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:05.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.017+0000 7f3116151700 1 --2- 172.21.15.77:0/1134489128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 0x7f3110102780 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f3104004940 tx=0x7f3104009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:05.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.017+0000 7f3116151700 1 -- 172.21.15.77:0/1134489128 shutdown_connections 2024-02-19T18:59:05.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.017+0000 7f3116151700 1 --2- 172.21.15.77:0/1134489128 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 0x7f3110102780 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.017+0000 7f3116151700 1 --2- 172.21.15.77:0/1134489128 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31100fda80 0x7f31100ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.017+0000 7f3116151700 1 -- 172.21.15.77:0/1134489128 >> 172.21.15.77:0/1134489128 conn(0x7f31100f74e0 msgr2=0x7f31100f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.017+0000 7f3116151700 1 -- 172.21.15.77:0/1134489128 shutdown_connections 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.017+0000 7f3116151700 1 -- 172.21.15.77:0/1134489128 wait complete. 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f3116151700 1 Processor -- start 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f3116151700 1 -- start start 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f3116151700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31100fda80 0x7f311010d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f3116151700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 0x7f311010de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f3116151700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311010e480 con 0x7f31101003a0 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f3116151700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311010ed20 con 0x7f31100fda80 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f310effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 0x7f311010de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f310effd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 0x7f311010de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:60634/0 (socket says 172.21.15.77:60634) 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f310effd700 1 -- 172.21.15.77:0/3292449704 learned_addr learned my addr 172.21.15.77:0/3292449704 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.018+0000 7f310f7fe700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31100fda80 0x7f311010d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:05.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f310effd700 1 -- 172.21.15.77:0/3292449704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31100fda80 msgr2=0x7f311010d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f310effd700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31100fda80 0x7f311010d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f310effd700 1 -- 172.21.15.77:0/3292449704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3104005040 con 0x7f31101003a0 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f310f7fe700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31100fda80 0x7f311010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f310effd700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 0x7f311010de80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f31040093b0 tx=0x7f3104009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f30f7fff700 1 -- 172.21.15.77:0/3292449704 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f310400e040 con 0x7f31101003a0 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f311010efa0 con 0x7f31101003a0 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f30f7fff700 1 -- 172.21.15.77:0/3292449704 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f310400bcc0 con 0x7f31101003a0 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f30f7fff700 1 -- 172.21.15.77:0/3292449704 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f310401c630 con 0x7f31101003a0 2024-02-19T18:59:05.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.019+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f311010f4e0 con 0x7f31101003a0 2024-02-19T18:59:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.020+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f311004bed0 con 0x7f31101003a0 2024-02-19T18:59:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.021+0000 7f30f7fff700 1 -- 172.21.15.77:0/3292449704 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f3104013980 con 0x7f31101003a0 2024-02-19T18:59:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.021+0000 7f30f7fff700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f30f80692a0 0x7f30f806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.021+0000 7f30f7fff700 1 -- 172.21.15.77:0/3292449704 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3104088500 con 0x7f31101003a0 2024-02-19T18:59:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.021+0000 7f310f7fe700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f30f80692a0 0x7f30f806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:05.040 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.022+0000 7f310f7fe700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f30f80692a0 0x7f30f806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3108007dc0 tx=0x7f3108009da0 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:05.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.026+0000 7f30f7fff700 1 -- 172.21.15.77:0/3292449704 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f31040582c0 con 0x7f31101003a0 2024-02-19T18:59:05.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.215+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f3110105a10 con 0x7f31101003a0 2024-02-19T18:59:05.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.215+0000 7f30f7fff700 1 -- 172.21.15.77:0/3292449704 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f3104057e50 con 0x7f31101003a0 2024-02-19T18:59:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f30f80692a0 msgr2=0x7f30f806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f30f80692a0 0x7f30f806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3108007dc0 tx=0x7f3108009da0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 msgr2=0x7f311010de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 0x7f311010de80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f31040093b0 tx=0x7f3104009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 shutdown_connections 2024-02-19T18:59:05.235 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f30f80692a0 0x7f30f806b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31100fda80 0x7f311010d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 --2- 172.21.15.77:0/3292449704 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31101003a0 0x7f311010de80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 >> 172.21.15.77:0/3292449704 conn(0x7f31100f74e0 msgr2=0x7f31100f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 shutdown_connections 2024-02-19T18:59:05.236 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:05.217+0000 7f3116151700 1 -- 172.21.15.77:0/3292449704 wait complete. 2024-02-19T18:59:05.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:05 smithi077 ceph-mon[29247]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-19T18:59:05.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:05 smithi104 ceph-mon[42320]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-19T18:59:06.038 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-02-19T18:59:06.311 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:59:07.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:07 smithi077 ceph-mon[29247]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-19T18:59:07.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:07 smithi104 ceph-mon[42320]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-19T18:59:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.121+0000 7ff86047b700 1 -- 172.21.15.77:0/1631962251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fcfc0 msgr2=0x7ff8580fd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.121+0000 7ff86047b700 1 --2- 172.21.15.77:0/1631962251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fcfc0 0x7ff8580fd430 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff848004940 tx=0x7ff848009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.121+0000 7ff86047b700 1 -- 172.21.15.77:0/1631962251 shutdown_connections 2024-02-19T18:59:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.121+0000 7ff86047b700 1 --2- 172.21.15.77:0/1631962251 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fcfc0 0x7ff8580fd430 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.121+0000 7ff86047b700 1 --2- 172.21.15.77:0/1631962251 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8580fbe50 0x7ff8580fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.121+0000 7ff86047b700 1 -- 172.21.15.77:0/1631962251 >> 172.21.15.77:0/1631962251 conn(0x7ff8580f7510 msgr2=0x7ff8580f9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.121+0000 7ff86047b700 1 -- 172.21.15.77:0/1631962251 shutdown_connections 2024-02-19T18:59:08.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.121+0000 7ff86047b700 1 -- 172.21.15.77:0/1631962251 wait complete. 2024-02-19T18:59:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff86047b700 1 Processor -- start 2024-02-19T18:59:08.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff86047b700 1 -- start start 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff86047b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fbe50 0x7ff85810d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff86047b700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8580fcfc0 0x7ff85810de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff86047b700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85810e490 con 0x7ff8580fbe50 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff86047b700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85810ed30 con 0x7ff8580fcfc0 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff85e217700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fbe50 0x7ff85810d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff85e217700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fbe50 0x7ff85810d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:37834/0 (socket says 172.21.15.77:37834) 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.122+0000 7ff85e217700 1 -- 172.21.15.77:0/4077023859 learned_addr learned my addr 172.21.15.77:0/4077023859 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff85da16700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8580fcfc0 0x7ff85810de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff85e217700 1 -- 172.21.15.77:0/4077023859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8580fcfc0 msgr2=0x7ff85810de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff85e217700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8580fcfc0 0x7ff85810de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff85e217700 1 -- 172.21.15.77:0/4077023859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848005040 con 0x7ff8580fbe50 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff85e217700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fbe50 0x7ff85810d950 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff854002a50 tx=0x7ff8540046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff84effd700 1 -- 172.21.15.77:0/4077023859 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff854007cc0 con 0x7ff8580fbe50 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff85810f010 con 0x7ff8580fbe50 2024-02-19T18:59:08.142 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff84effd700 1 -- 172.21.15.77:0/4077023859 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff85400f040 con 0x7ff8580fbe50 2024-02-19T18:59:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff84effd700 1 -- 172.21.15.77:0/4077023859 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff85400da30 con 0x7ff8580fbe50 2024-02-19T18:59:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.123+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff85810f550 con 0x7ff8580fbe50 2024-02-19T18:59:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.125+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85804bed0 con 0x7ff8580fbe50 2024-02-19T18:59:08.143 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.125+0000 7ff84effd700 1 -- 172.21.15.77:0/4077023859 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7ff85400db90 con 0x7ff8580fbe50 2024-02-19T18:59:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.125+0000 7ff84effd700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8440692a0 0x7ff84406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.125+0000 7ff84effd700 1 -- 172.21.15.77:0/4077023859 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff854086f20 con 0x7ff8580fbe50 2024-02-19T18:59:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.126+0000 7ff85da16700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8440692a0 0x7ff84406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:08.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.126+0000 7ff85da16700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8440692a0 0x7ff84406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff84800b3b0 tx=0x7ff848009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:08.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.130+0000 7ff84effd700 1 -- 172.21.15.77:0/4077023859 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7ff854056c60 con 0x7ff8580fbe50 2024-02-19T18:59:08.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:08.344+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7ff858066660 con 0x7ff8440692a0 2024-02-19T18:59:09.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:08 smithi077 ceph-mon[29247]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-19T18:59:09.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:08 smithi077 ceph-mon[29247]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:09.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:08 smithi104 ceph-mon[42320]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 767 B/s wr, 0 op/s 2024-02-19T18:59:09.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:08 smithi104 ceph-mon[42320]: from='client.14600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:11.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:11 smithi077 ceph-mon[29247]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:11.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:11 smithi104 ceph-mon[42320]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:13.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:13 smithi077 ceph-mon[29247]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:13.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:13 smithi104 ceph-mon[42320]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:15.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:15 smithi077 ceph-mon[29247]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-19T18:59:15.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:15 smithi104 ceph-mon[42320]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-19T18:59:17.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:17 smithi077 ceph-mon[29247]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:17.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:17 smithi104 ceph-mon[42320]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:19.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:18 smithi077 ceph-mon[29247]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:19.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:18 smithi104 ceph-mon[42320]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:20.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:20.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:21.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:20 smithi077 ceph-mon[29247]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:20 smithi104 ceph-mon[42320]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:23.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:23 smithi077 ceph-mon[29247]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:23.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:23 smithi104 ceph-mon[42320]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:25.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:25 smithi077 ceph-mon[29247]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:25.562 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:25 smithi104 ceph-mon[42320]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:27.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:27 smithi077 ceph-mon[29247]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:27.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:27 smithi104 ceph-mon[42320]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:29.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:28 smithi077 ceph-mon[29247]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:29.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:28 smithi104 ceph-mon[42320]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:31.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:31 smithi077 ceph-mon[29247]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:31.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:31 smithi104 ceph-mon[42320]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:33.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:33 smithi077 ceph-mon[29247]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:33.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:33 smithi104 ceph-mon[42320]: pgmap v213: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:34.350 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:34.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:35.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.251+0000 7ff84effd700 1 -- 172.21.15.77:0/4077023859 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7ff858066660 con 0x7ff8440692a0 2024-02-19T18:59:35.270 INFO:teuthology.orchestra.run.smithi077.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8440692a0 msgr2=0x7ff84406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8440692a0 0x7ff84406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff84800b3b0 tx=0x7ff848009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fbe50 msgr2=0x7ff85810d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fbe50 0x7ff85810d950 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff854002a50 tx=0x7ff8540046b0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 shutdown_connections 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7ff8440692a0 0x7ff84406b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff8580fbe50 0x7ff85810d950 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 --2- 172.21.15.77:0/4077023859 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff8580fcfc0 0x7ff85810de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 >> 172.21.15.77:0/4077023859 conn(0x7ff8580f7510 msgr2=0x7ff858065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 shutdown_connections 2024-02-19T18:59:35.272 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:35.253+0000 7ff86047b700 1 -- 172.21.15.77:0/4077023859 wait complete. 2024-02-19T18:59:35.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:35 smithi077 ceph-mon[29247]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:35.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:35 smithi104 ceph-mon[42320]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:36.006 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-02-19T18:59:36.274 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T18:59:36.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:36 smithi077 ceph-mon[29247]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:59:36.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:36 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:59:36.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:36 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:36.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:36 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:36.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:36 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:36.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:36 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:59:36.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:36 smithi104 ceph-mon[42320]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:59:36.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:36 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:59:36.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:36 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:36.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:36 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T18:59:36.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:36 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T18:59:36.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:36 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:59:37.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:37 smithi077 ceph-mon[29247]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:37.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:37 smithi077 ceph-mon[29247]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:59:37.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:37 smithi104 ceph-mon[42320]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 426 B/s wr, 0 op/s 2024-02-19T18:59:37.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:37 smithi104 ceph-mon[42320]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 -- 172.21.15.77:0/2048346262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fcfb0 msgr2=0x7f43a80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 --2- 172.21.15.77:0/2048346262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fcfb0 0x7f43a80fd420 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f4390004940 tx=0x7f4390009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 -- 172.21.15.77:0/2048346262 shutdown_connections 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 --2- 172.21.15.77:0/2048346262 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fcfb0 0x7f43a80fd420 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 --2- 172.21.15.77:0/2048346262 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a80fbe40 0x7f43a80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 -- 172.21.15.77:0/2048346262 >> 172.21.15.77:0/2048346262 conn(0x7f43a80f7520 msgr2=0x7f43a80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 -- 172.21.15.77:0/2048346262 shutdown_connections 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 -- 172.21.15.77:0/2048346262 wait complete. 2024-02-19T18:59:39.014 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 Processor -- start 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.994+0000 7f43ade10700 1 -- start start 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43ade10700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fbe40 0x7f43a81105f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43ade10700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a80fcfb0 0x7f43a8112b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43ade10700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a8066de0 con 0x7f43a80fbe40 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43ade10700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a8066f20 con 0x7f43a80fcfb0 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fbe40 0x7f43a81105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43a77fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fbe40 0x7f43a81105f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49164/0 (socket says 172.21.15.77:49164) 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43a77fe700 1 -- 172.21.15.77:0/4195846769 learned_addr learned my addr 172.21.15.77:0/4195846769 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43a6ffd700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a80fcfb0 0x7f43a8112b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43a77fe700 1 -- 172.21.15.77:0/4195846769 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a80fcfb0 msgr2=0x7f43a8112b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43a77fe700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a80fcfb0 0x7f43a8112b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43a77fe700 1 -- 172.21.15.77:0/4195846769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4390005040 con 0x7f43a80fbe40 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43a77fe700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fbe40 0x7f43a81105f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b790 tx=0x7f43a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:39.015 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f439ffff700 1 -- 172.21.15.77:0/4195846769 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43a000f770 con 0x7f43a80fbe40 2024-02-19T18:59:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a8113220 con 0x7f43a80fbe40 2024-02-19T18:59:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f439ffff700 1 -- 172.21.15.77:0/4195846769 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f43a000fda0 con 0x7f43a80fbe40 2024-02-19T18:59:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.995+0000 7f439ffff700 1 -- 172.21.15.77:0/4195846769 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f43a000d310 con 0x7f43a80fbe40 2024-02-19T18:59:39.016 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.996+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a8113760 con 0x7f43a80fbe40 2024-02-19T18:59:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.996+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a804bed0 con 0x7f43a80fbe40 2024-02-19T18:59:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.997+0000 7f439ffff700 1 -- 172.21.15.77:0/4195846769 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f43a0010420 con 0x7f43a80fbe40 2024-02-19T18:59:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.997+0000 7f439ffff700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f43940692a0 0x7f439406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.997+0000 7f439ffff700 1 -- 172.21.15.77:0/4195846769 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f43a00871b0 con 0x7f43a80fbe40 2024-02-19T18:59:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.997+0000 7f43a6ffd700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f43940692a0 0x7f439406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:38.998+0000 7f43a6ffd700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f43940692a0 0x7f439406b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4390004f90 tx=0x7f4390009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:39.020 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.002+0000 7f439ffff700 1 -- 172.21.15.77:0/4195846769 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f43a00534e0 con 0x7f43a80fbe40 2024-02-19T18:59:39.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.205+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f43a8101990 con 0x7f43940692a0 2024-02-19T18:59:39.224 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:38 smithi077 ceph-mon[29247]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T18:59:39.224 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.206+0000 7f439ffff700 1 -- 172.21.15.77:0/4195846769 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f43a8101990 con 0x7f43940692a0 2024-02-19T18:59:39.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f43940692a0 msgr2=0x7f439406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f43940692a0 0x7f439406b750 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4390004f90 tx=0x7f4390009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fbe40 msgr2=0x7f43a81105f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fbe40 0x7f43a81105f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f43a000b790 tx=0x7f43a0009250 comp rx=0 tx=0).stop 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 shutdown_connections 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f43940692a0 0x7f439406b750 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f43a80fbe40 0x7f43a81105f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 --2- 172.21.15.77:0/4195846769 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f43a80fcfb0 0x7f43a8112b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 >> 172.21.15.77:0/4195846769 conn(0x7f43a80f7520 msgr2=0x7f43a8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 shutdown_connections 2024-02-19T18:59:39.227 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.208+0000 7f43ade10700 1 -- 172.21.15.77:0/4195846769 wait complete. 2024-02-19T18:59:39.237 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T18:59:39.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:38 smithi104 ceph-mon[42320]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T18:59:39.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 -- 172.21.15.77:0/4208056931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 msgr2=0x7f00f80fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 --2- 172.21.15.77:0/4208056931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 0x7f00f80fd420 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f00ec004940 tx=0x7f00ec009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:39.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 -- 172.21.15.77:0/4208056931 shutdown_connections 2024-02-19T18:59:39.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 --2- 172.21.15.77:0/4208056931 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 0x7f00f80fd420 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 --2- 172.21.15.77:0/4208056931 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f80fbe40 0x7f00f80fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 -- 172.21.15.77:0/4208056931 >> 172.21.15.77:0/4208056931 conn(0x7f00f80f7500 msgr2=0x7f00f80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:39.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 -- 172.21.15.77:0/4208056931 shutdown_connections 2024-02-19T18:59:39.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 -- 172.21.15.77:0/4208056931 wait complete. 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 Processor -- start 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.313+0000 7f00fe077700 1 -- start start 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00fe077700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f80fbe40 0x7f00f8110660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00fe077700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 0x7f00f8112bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00fe077700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8066cf0 con 0x7f00f80fcfb0 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00fe077700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8066e30 con 0x7f00f80fbe40 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 0x7f00f8112bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00f6ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 0x7f00f8112bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49184/0 (socket says 172.21.15.77:49184) 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00f6ffd700 1 -- 172.21.15.77:0/302529815 learned_addr learned my addr 172.21.15.77:0/302529815 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:59:39.333 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00f77fe700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f80fbe40 0x7f00f8110660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00f6ffd700 1 -- 172.21.15.77:0/302529815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f80fbe40 msgr2=0x7f00f8110660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00f6ffd700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f80fbe40 0x7f00f8110660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00f6ffd700 1 -- 172.21.15.77:0/302529815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ec005040 con 0x7f00f80fcfb0 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00f6ffd700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 0x7f00f8112bb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0093b0 tx=0x7f00ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00dffff700 1 -- 172.21.15.77:0/302529815 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00ec00e040 con 0x7f00f80fcfb0 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00dffff700 1 -- 172.21.15.77:0/302529815 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f00ec00bcc0 con 0x7f00f80fcfb0 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f81131b0 con 0x7f00f80fcfb0 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00dffff700 1 -- 172.21.15.77:0/302529815 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f00ec01c630 con 0x7f00f80fcfb0 2024-02-19T18:59:39.334 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.314+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f81136f0 con 0x7f00f80fcfb0 2024-02-19T18:59:39.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.315+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f804bed0 con 0x7f00f80fcfb0 2024-02-19T18:59:39.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.316+0000 7f00dffff700 1 -- 172.21.15.77:0/302529815 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f00ec013980 con 0x7f00f80fcfb0 2024-02-19T18:59:39.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.316+0000 7f00dffff700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f00e00692a0 0x7f00e006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.316+0000 7f00dffff700 1 -- 172.21.15.77:0/302529815 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f00ec088540 con 0x7f00f80fcfb0 2024-02-19T18:59:39.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.316+0000 7f00f77fe700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f00e00692a0 0x7f00e006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.317+0000 7f00f77fe700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f00e00692a0 0x7f00e006b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f00f8066970 tx=0x7f00e800b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:39.339 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.321+0000 7f00dffff700 1 -- 172.21.15.77:0/302529815 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f00ec054870 con 0x7f00f80fcfb0 2024-02-19T18:59:39.554 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.536+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f00f8101990 con 0x7f00e00692a0 2024-02-19T18:59:39.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.537+0000 7f00dffff700 1 -- 172.21.15.77:0/302529815 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+341 (secure 0 0 0) 0x7f00f8101990 con 0x7f00e00692a0 2024-02-19T18:59:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.539+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f00e00692a0 msgr2=0x7f00e006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.539+0000 7f00fe077700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f00e00692a0 0x7f00e006b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f00f8066970 tx=0x7f00e800b040 comp rx=0 tx=0).stop 2024-02-19T18:59:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.539+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 msgr2=0x7f00f8112bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.539+0000 7f00fe077700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 0x7f00f8112bb0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0093b0 tx=0x7f00ec009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.540+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 shutdown_connections 2024-02-19T18:59:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.540+0000 7f00fe077700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f00e00692a0 0x7f00e006b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.540+0000 7f00fe077700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f00f80fbe40 0x7f00f8110660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.540+0000 7f00fe077700 1 --2- 172.21.15.77:0/302529815 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f00f80fcfb0 0x7f00f8112bb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.540+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 >> 172.21.15.77:0/302529815 conn(0x7f00f80f7500 msgr2=0x7f00f8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:39.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.540+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 shutdown_connections 2024-02-19T18:59:39.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.540+0000 7f00fe077700 1 -- 172.21.15.77:0/302529815 wait complete. 2024-02-19T18:59:39.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 -- 172.21.15.77:0/3050100633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c1003a0 msgr2=0x7f332c102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 --2- 172.21.15.77:0/3050100633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c1003a0 0x7f332c102780 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f3314004940 tx=0x7f3314009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:39.660 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 -- 172.21.15.77:0/3050100633 shutdown_connections 2024-02-19T18:59:39.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 --2- 172.21.15.77:0/3050100633 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c1003a0 0x7f332c102780 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 --2- 172.21.15.77:0/3050100633 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332c0fda80 0x7f332c0ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 -- 172.21.15.77:0/3050100633 >> 172.21.15.77:0/3050100633 conn(0x7f332c0f74e0 msgr2=0x7f332c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:39.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 -- 172.21.15.77:0/3050100633 shutdown_connections 2024-02-19T18:59:39.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 -- 172.21.15.77:0/3050100633 wait complete. 2024-02-19T18:59:39.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 Processor -- start 2024-02-19T18:59:39.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.642+0000 7f3331db7700 1 -- start start 2024-02-19T18:59:39.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3331db7700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c0fda80 0x7f332c0fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3331db7700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332c1003a0 0x7f332c0fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3331db7700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0fbf90 con 0x7f332c0fda80 2024-02-19T18:59:39.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3331db7700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0fc100 con 0x7f332c1003a0 2024-02-19T18:59:39.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f332b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c0fda80 0x7f332c0fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f332b7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c0fda80 0x7f332c0fd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49204/0 (socket says 172.21.15.77:49204) 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f332b7fe700 1 -- 172.21.15.77:0/455436211 learned_addr learned my addr 172.21.15.77:0/455436211 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f332affd700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332c1003a0 0x7f332c0fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f332b7fe700 1 -- 172.21.15.77:0/455436211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332c1003a0 msgr2=0x7f332c0fb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f332b7fe700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332c1003a0 0x7f332c0fb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f332b7fe700 1 -- 172.21.15.77:0/455436211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3314005040 con 0x7f332c0fda80 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f332b7fe700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c0fda80 0x7f332c0fd510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3324001d80 tx=0x7f3324009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3323fff700 1 -- 172.21.15.77:0/455436211 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3324007d10 con 0x7f332c0fda80 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332c0fc3e0 con 0x7f332c0fda80 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3323fff700 1 -- 172.21.15.77:0/455436211 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3324001e80 con 0x7f332c0fda80 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3323fff700 1 -- 172.21.15.77:0/455436211 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3324014640 con 0x7f332c0fda80 2024-02-19T18:59:39.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.643+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332c115970 con 0x7f332c0fda80 2024-02-19T18:59:39.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.644+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c04bed0 con 0x7f332c0fda80 2024-02-19T18:59:39.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.645+0000 7f3323fff700 1 -- 172.21.15.77:0/455436211 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f332400e960 con 0x7f332c0fda80 2024-02-19T18:59:39.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.645+0000 7f3323fff700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f33180692a0 0x7f331806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.645+0000 7f3323fff700 1 -- 172.21.15.77:0/455436211 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3324086f00 con 0x7f332c0fda80 2024-02-19T18:59:39.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.645+0000 7f332affd700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f33180692a0 0x7f331806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.645+0000 7f332affd700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f33180692a0 0x7f331806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3314004f90 tx=0x7f3314009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:39.667 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.649+0000 7f3323fff700 1 -- 172.21.15.77:0/455436211 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f332404dbd0 con 0x7f332c0fda80 2024-02-19T18:59:39.872 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.853+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f332c101000 con 0x7f33180692a0 2024-02-19T18:59:39.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.862+0000 7f3323fff700 1 -- 172.21.15.77:0/455436211 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7f332c101000 con 0x7f33180692a0 2024-02-19T18:59:39.881 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T18:59:39.881 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (5m) 52s ago 8m 26.7M - 0.25.0 c8568f914cd2 1e676c9ed794 2024-02-19T18:59:39.881 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (8m) 52s ago 8m 6656k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T18:59:39.881 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (5m) 56s ago 5m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T18:59:39.881 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (5m) 52s ago 6m 86.8M - 9.4.7 954c08fa6188 2cb428963663 2024-02-19T18:59:39.881 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (71s) 52s ago 70s 12.2M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T18:59:39.881 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (76s) 52s ago 76s 17.1M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (73s) 56s ago 73s 12.0M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (68s) 56s ago 68s 11.1M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:9283 running (9m) 52s ago 9m 480M - 17.2.7-521-gc80e7168 10b0c032c913 78483a70884c 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283 running (5m) 56s ago 5m 435M - 17.2.7-521-gc80e7168 10b0c032c913 5693aa48a572 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (9m) 52s ago 9m 46.6M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (5m) 56s ago 5m 35.5M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (8m) 52s ago 8m 14.2M - 1.5.0 0da6a335fe13 18fa31ab4762 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (5m) 56s ago 5m 15.5M - 1.5.0 0da6a335fe13 5ec125749863 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (4m) 52s ago 4m 46.4M 4288M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (4m) 52s ago 4m 44.9M 4288M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (4m) 52s ago 3m 43.9M 4288M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (3m) 56s ago 3m 44.5M 5312M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (3m) 56s ago 3m 44.1M 5312M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (2m) 56s ago 2m 44.0M 5312M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T18:59:39.882 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (5m) 52s ago 6m 55.1M - 2.43.0 a07b618ecd1d 6dc30fbe4b32 2024-02-19T18:59:39.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.864+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f33180692a0 msgr2=0x7f331806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.864+0000 7f3331db7700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f33180692a0 0x7f331806b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3314004f90 tx=0x7f3314009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:39.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.864+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c0fda80 msgr2=0x7f332c0fd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.864+0000 7f3331db7700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c0fda80 0x7f332c0fd510 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3324001d80 tx=0x7f3324009ec0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.865+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 shutdown_connections 2024-02-19T18:59:39.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.865+0000 7f3331db7700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f33180692a0 0x7f331806b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.865+0000 7f3331db7700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f332c0fda80 0x7f332c0fd510 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.883 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.865+0000 7f3331db7700 1 --2- 172.21.15.77:0/455436211 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f332c1003a0 0x7f332c0fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.865+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 >> 172.21.15.77:0/455436211 conn(0x7f332c0f74e0 msgr2=0x7f332c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:39.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.865+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 shutdown_connections 2024-02-19T18:59:39.884 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.865+0000 7f3331db7700 1 -- 172.21.15.77:0/455436211 wait complete. 2024-02-19T18:59:39.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.970+0000 7fb4af496700 1 -- 172.21.15.77:0/1465928907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 msgr2=0x7fb4a80fd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.970+0000 7fb4af496700 1 --2- 172.21.15.77:0/1465928907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 0x7fb4a80fd920 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4004940 tx=0x7fb4a4009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:39.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.970+0000 7fb4af496700 1 -- 172.21.15.77:0/1465928907 shutdown_connections 2024-02-19T18:59:39.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.970+0000 7fb4af496700 1 --2- 172.21.15.77:0/1465928907 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a80fde60 0x7fb4a81047b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.970+0000 7fb4af496700 1 --2- 172.21.15.77:0/1465928907 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 0x7fb4a80fd920 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.970+0000 7fb4af496700 1 -- 172.21.15.77:0/1465928907 >> 172.21.15.77:0/1465928907 conn(0x7fb4a80f6ca0 msgr2=0x7fb4a80f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.970+0000 7fb4af496700 1 -- 172.21.15.77:0/1465928907 shutdown_connections 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.970+0000 7fb4af496700 1 -- 172.21.15.77:0/1465928907 wait complete. 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4af496700 1 Processor -- start 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4af496700 1 -- start start 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4af496700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 0x7fb4a810fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4af496700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a80fde60 0x7fb4a81100c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4af496700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81106c0 con 0x7fb4a80fd510 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4af496700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a8110f60 con 0x7fb4a80fde60 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4ad232700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 0x7fb4a810fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4ad232700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 0x7fb4a810fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49214/0 (socket says 172.21.15.77:49214) 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4ad232700 1 -- 172.21.15.77:0/3864419684 learned_addr learned my addr 172.21.15.77:0/3864419684 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:59:39.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4aca31700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a80fde60 0x7fb4a81100c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4ad232700 1 -- 172.21.15.77:0/3864419684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a80fde60 msgr2=0x7fb4a81100c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4ad232700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a80fde60 0x7fb4a81100c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.971+0000 7fb4ad232700 1 -- 172.21.15.77:0/3864419684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a4005040 con 0x7fb4a80fd510 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.972+0000 7fb4ad232700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 0x7fb4a810fb80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009b30 tx=0x7fb4a4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.972+0000 7fb49dffb700 1 -- 172.21.15.77:0/3864419684 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4a400e070 con 0x7fb4a80fd510 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.972+0000 7fb49dffb700 1 -- 172.21.15.77:0/3864419684 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4a4009430 con 0x7fb4a80fd510 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.972+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a81111e0 con 0x7fb4a80fd510 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.972+0000 7fb49dffb700 1 -- 172.21.15.77:0/3864419684 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb4a400b410 con 0x7fb4a80fd510 2024-02-19T18:59:39.992 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.972+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a81116c0 con 0x7fb4a80fd510 2024-02-19T18:59:39.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.973+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a804bed0 con 0x7fb4a80fd510 2024-02-19T18:59:39.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.973+0000 7fb49dffb700 1 -- 172.21.15.77:0/3864419684 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fb4a401c870 con 0x7fb4a80fd510 2024-02-19T18:59:39.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.974+0000 7fb49dffb700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb4940692a0 0x7fb49406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:39.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.974+0000 7fb49dffb700 1 -- 172.21.15.77:0/3864419684 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb4a40886a0 con 0x7fb4a80fd510 2024-02-19T18:59:39.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.974+0000 7fb4aca31700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb4940692a0 0x7fb49406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:39.993 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.974+0000 7fb4aca31700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb4940692a0 0x7fb49406b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb498009b80 tx=0x7fb49800b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:39.996 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:39.978+0000 7fb49dffb700 1 -- 172.21.15.77:0/3864419684 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fb4a4058500 con 0x7fb4a80fd510 2024-02-19T18:59:40.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.251+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7fb4a8111d10 con 0x7fb4a80fd510 2024-02-19T18:59:40.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.252+0000 7fb49dffb700 1 -- 172.21.15.77:0/3864419684 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7fb4a4086e60 con 0x7fb4a80fd510 2024-02-19T18:59:40.270 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T18:59:40.270 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 14 2024-02-19T18:59:40.271 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T18:59:40.272 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb4940692a0 msgr2=0x7fb49406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb4940692a0 0x7fb49406b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb498009b80 tx=0x7fb49800b040 comp rx=0 tx=0).stop 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 msgr2=0x7fb4a810fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 0x7fb4a810fb80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4009b30 tx=0x7fb4a4009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 shutdown_connections 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fb4940692a0 0x7fb49406b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fb4a80fd510 0x7fb4a810fb80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 --2- 172.21.15.77:0/3864419684 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fb4a80fde60 0x7fb4a81100c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 >> 172.21.15.77:0/3864419684 conn(0x7fb4a80f6ca0 msgr2=0x7fb4a80f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:40.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 shutdown_connections 2024-02-19T18:59:40.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.254+0000 7fb4af496700 1 -- 172.21.15.77:0/3864419684 wait complete. 2024-02-19T18:59:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.357+0000 7fa8e92e8700 1 -- 172.21.15.77:0/555039048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 msgr2=0x7fa8e40fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.357+0000 7fa8e92e8700 1 --2- 172.21.15.77:0/555039048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 0x7fa8e40fe550 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8004940 tx=0x7fa8d8009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.357+0000 7fa8e92e8700 1 -- 172.21.15.77:0/555039048 shutdown_connections 2024-02-19T18:59:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.357+0000 7fa8e92e8700 1 --2- 172.21.15.77:0/555039048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 0x7fa8e40fe550 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.357+0000 7fa8e92e8700 1 --2- 172.21.15.77:0/555039048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e40fcef0 0x7fa8e40fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.357+0000 7fa8e92e8700 1 -- 172.21.15.77:0/555039048 >> 172.21.15.77:0/555039048 conn(0x7fa8e40f8620 msgr2=0x7fa8e40faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.357+0000 7fa8e92e8700 1 -- 172.21.15.77:0/555039048 shutdown_connections 2024-02-19T18:59:40.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.357+0000 7fa8e92e8700 1 -- 172.21.15.77:0/555039048 wait complete. 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e92e8700 1 Processor -- start 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e92e8700 1 -- start start 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e92e8700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e40fcef0 0x7fa8e4071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e92e8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 0x7fa8e40702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e92e8700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e40708e0 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e92e8700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e4070a50 con 0x7fa8e40fcef0 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e27fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 0x7fa8e40702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e27fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 0x7fa8e40702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49224/0 (socket says 172.21.15.77:49224) 2024-02-19T18:59:40.378 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.358+0000 7fa8e27fc700 1 -- 172.21.15.77:0/4164527415 learned_addr learned my addr 172.21.15.77:0/4164527415 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8e2ffd700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e40fcef0 0x7fa8e4071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8e27fc700 1 -- 172.21.15.77:0/4164527415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e40fcef0 msgr2=0x7fa8e4071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8e27fc700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e40fcef0 0x7fa8e4071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8e27fc700 1 -- 172.21.15.77:0/4164527415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8005040 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8e2ffd700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e40fcef0 0x7fa8e4071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8e27fc700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 0x7fa8e40702e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009fd0 tx=0x7fa8d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8cb7fe700 1 -- 172.21.15.77:0/4164527415 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d800e040 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8cb7fe700 1 -- 172.21.15.77:0/4164527415 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa8d800bcc0 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e4070cd0 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8cb7fe700 1 -- 172.21.15.77:0/4164527415 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d801c630 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.359+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e4115990 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.379 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.360+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e404bed0 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.360+0000 7fa8cb7fe700 1 -- 172.21.15.77:0/4164527415 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fa8d8013980 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.361+0000 7fa8cb7fe700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa8cc0692b0 0x7fa8cc06b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T18:59:40.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.361+0000 7fa8cb7fe700 1 -- 172.21.15.77:0/4164527415 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fa8d8088650 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.380 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.361+0000 7fa8e2ffd700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa8cc0692b0 0x7fa8cc06b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T18:59:40.381 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.361+0000 7fa8e2ffd700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa8cc0692b0 0x7fa8cc06b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4071bf0 tx=0x7fa8d4009040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T18:59:40.383 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.365+0000 7fa8cb7fe700 1 -- 172.21.15.77:0/4164527415 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fa8d8058410 con 0x7fa8e40fe0e0 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: Upgrade: Target is version 19.0.0-1450-gf78a58c0 (unknown) 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, digests ['quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526'] 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: Upgrade: Need to upgrade myself (mgr.smithi077.yiiqpd) 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:40.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:40 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3864419684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T18:59:40.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.579+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8e4102ac0 con 0x7fa8cc0692b0 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.580+0000 7fa8cb7fe700 1 -- 172.21.15.77:0/4164527415 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fa8e4102ac0 con 0x7fa8cc0692b0 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "0/2 daemons upgraded", 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 image on host smithi104", 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T18:59:40.599 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T18:59:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.582+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa8cc0692b0 msgr2=0x7fa8cc06b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.582+0000 7fa8e92e8700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa8cc0692b0 0x7fa8cc06b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4071bf0 tx=0x7fa8d4009040 comp rx=0 tx=0).stop 2024-02-19T18:59:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.582+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 msgr2=0x7fa8e40702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T18:59:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.582+0000 7fa8e92e8700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 0x7fa8e40702e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fa8d8009fd0 tx=0x7fa8d8009e30 comp rx=0 tx=0).stop 2024-02-19T18:59:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.582+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 shutdown_connections 2024-02-19T18:59:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.582+0000 7fa8e92e8700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fa8cc0692b0 0x7fa8cc06b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.601 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.582+0000 7fa8e92e8700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa8e40fcef0 0x7fa8e4071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.582+0000 7fa8e92e8700 1 --2- 172.21.15.77:0/4164527415 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa8e40fe0e0 0x7fa8e40702e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T18:59:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.583+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 >> 172.21.15.77:0/4164527415 conn(0x7fa8e40f8620 msgr2=0x7fa8e41013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T18:59:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.583+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 shutdown_connections 2024-02-19T18:59:40.602 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T18:59:40.583+0000 7fa8e92e8700 1 -- 172.21.15.77:0/4164527415 wait complete. 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: Upgrade: Target is version 19.0.0-1450-gf78a58c0 (unknown) 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526, digests ['quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526'] 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: Upgrade: Need to upgrade myself (mgr.smithi077.yiiqpd) 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:40.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:40 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3864419684' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T18:59:41.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:41 smithi077 ceph-mon[29247]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 on smithi104 2024-02-19T18:59:41.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:41 smithi077 ceph-mon[29247]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:41.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:41 smithi077 ceph-mon[29247]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T18:59:41.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:41 smithi077 ceph-mon[29247]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:41.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:41 smithi104 ceph-mon[42320]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 on smithi104 2024-02-19T18:59:41.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:41 smithi104 ceph-mon[42320]: from='client.14612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:41.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:41 smithi104 ceph-mon[42320]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T18:59:41.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:41 smithi104 ceph-mon[42320]: from='client.14620 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T18:59:43.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:43 smithi077 ceph-mon[29247]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T18:59:43.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:43 smithi104 ceph-mon[42320]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 511 B/s wr, 0 op/s 2024-02-19T18:59:45.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:45 smithi077 ceph-mon[29247]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-19T18:59:45.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:45 smithi104 ceph-mon[42320]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-19T18:59:47.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:47 smithi077 ceph-mon[29247]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-19T18:59:47.812 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:47 smithi104 ceph-mon[42320]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-19T18:59:49.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:48 smithi077 ceph-mon[29247]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-19T18:59:49.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:48 smithi104 ceph-mon[42320]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 170 B/s wr, 0 op/s 2024-02-19T18:59:50.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:49 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:50.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:49 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T18:59:51.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:50 smithi077 ceph-mon[29247]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-19T18:59:51.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:50 smithi104 ceph-mon[42320]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-19T18:59:53.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:53 smithi077 ceph-mon[29247]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-19T18:59:53.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:53 smithi104 ceph-mon[42320]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-19T18:59:55.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:55 smithi077 ceph-mon[29247]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-19T18:59:55.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:55 smithi104 ceph-mon[42320]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 85 B/s wr, 0 op/s 2024-02-19T18:59:57.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:57 smithi077 ceph-mon[29247]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:57.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:57 smithi104 ceph-mon[42320]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:59.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 18:59:58 smithi077 ceph-mon[29247]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T18:59:59.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 18:59:58 smithi104 ceph-mon[42320]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:00.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:00 smithi077 ceph-mon[29247]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:00:00.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:00 smithi077 ceph-mon[29247]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:00:00.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:00 smithi077 ceph-mon[29247]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:00:00.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:00 smithi104 ceph-mon[42320]: Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T19:00:00.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:00 smithi104 ceph-mon[42320]: [WRN] FS_INLINE_DATA_DEPRECATED: 1 filesystem with deprecated feature inline_data 2024-02-19T19:00:00.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:00 smithi104 ceph-mon[42320]: fs cephfs has deprecated feature inline_data enabled. 2024-02-19T19:00:01.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:01 smithi104 ceph-mon[42320]: pgmap v227: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:01.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:01 smithi077 ceph-mon[29247]: pgmap v227: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:03.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:03 smithi077 ceph-mon[29247]: pgmap v228: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:03.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:03 smithi104 ceph-mon[42320]: pgmap v228: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:04.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:04 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:04.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:04 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:05.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:05 smithi077 ceph-mon[29247]: pgmap v229: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:05.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:05 smithi104 ceph-mon[42320]: pgmap v229: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:07.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:07 smithi077 ceph-mon[29247]: pgmap v230: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:07.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:07 smithi104 ceph-mon[42320]: pgmap v230: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:09.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:08 smithi077 ceph-mon[29247]: pgmap v231: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:09.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:08 smithi104 ceph-mon[42320]: pgmap v231: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:10.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 -- 172.21.15.77:0/1287567059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 msgr2=0x7f6f0c102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:10.721 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 --2- 172.21.15.77:0/1287567059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 0x7f6f0c102780 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6f00004940 tx=0x7f6f00009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 -- 172.21.15.77:0/1287567059 shutdown_connections 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 --2- 172.21.15.77:0/1287567059 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 0x7f6f0c102780 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 --2- 172.21.15.77:0/1287567059 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f0c0fda80 0x7f6f0c0ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 -- 172.21.15.77:0/1287567059 >> 172.21.15.77:0/1287567059 conn(0x7f6f0c0f74e0 msgr2=0x7f6f0c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 -- 172.21.15.77:0/1287567059 shutdown_connections 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 -- 172.21.15.77:0/1287567059 wait complete. 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.701+0000 7f6f11d93700 1 Processor -- start 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f11d93700 1 -- start start 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f11d93700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f0c0fda80 0x7f6f0c0fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f11d93700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 0x7f6f0c0fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f11d93700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c0fbf90 con 0x7f6f0c1003a0 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f11d93700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0c0fc100 con 0x7f6f0c0fda80 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f0affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 0x7f6f0c0fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f0affd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 0x7f6f0c0fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38156/0 (socket says 172.21.15.77:38156) 2024-02-19T19:00:10.722 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f0b7fe700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f0c0fda80 0x7f6f0c0fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f0affd700 1 -- 172.21.15.77:0/2473359596 learned_addr learned my addr 172.21.15.77:0/2473359596 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f0affd700 1 -- 172.21.15.77:0/2473359596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f0c0fda80 msgr2=0x7f6f0c0fd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f0affd700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f0c0fda80 0x7f6f0c0fd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f0affd700 1 -- 172.21.15.77:0/2473359596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00005040 con 0x7f6f0c1003a0 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.702+0000 7f6f0b7fe700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f0c0fda80 0x7f6f0c0fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.703+0000 7f6f0affd700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 0x7f6f0c0fb990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6f00000c00 tx=0x7f6f00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.703+0000 7f6ef3fff700 1 -- 172.21.15.77:0/2473359596 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f00021070 con 0x7f6f0c1003a0 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.703+0000 7f6ef3fff700 1 -- 172.21.15.77:0/2473359596 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f0001c400 con 0x7f6f0c1003a0 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.703+0000 7f6ef3fff700 1 -- 172.21.15.77:0/2473359596 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f0000e070 con 0x7f6f0c1003a0 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.703+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f0c0fc380 con 0x7f6f0c1003a0 2024-02-19T19:00:10.723 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.703+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f0c115910 con 0x7f6f0c1003a0 2024-02-19T19:00:10.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.704+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0c107a50 con 0x7f6f0c1003a0 2024-02-19T19:00:10.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.704+0000 7f6ef3fff700 1 -- 172.21.15.77:0/2473359596 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f6f0001ca50 con 0x7f6f0c1003a0 2024-02-19T19:00:10.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.705+0000 7f6ef3fff700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ef40692a0 0x7f6ef406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:10.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.705+0000 7f6ef3fff700 1 -- 172.21.15.77:0/2473359596 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6f00088290 con 0x7f6f0c1003a0 2024-02-19T19:00:10.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.705+0000 7f6f0b7fe700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ef40692a0 0x7f6ef406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:10.727 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.705+0000 7f6f0b7fe700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ef40692a0 0x7f6ef406b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0fd2a0 tx=0x7f6efc00b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:10.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.709+0000 7f6ef3fff700 1 -- 172.21.15.77:0/2473359596 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f6f000580f0 con 0x7f6f0c1003a0 2024-02-19T19:00:10.938 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.919+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f0c02b310 con 0x7f6ef40692a0 2024-02-19T19:00:10.939 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.920+0000 7f6ef3fff700 1 -- 172.21.15.77:0/2473359596 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f6f0c02b310 con 0x7f6ef40692a0 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ef40692a0 msgr2=0x7f6ef406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ef40692a0 0x7f6ef406b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c0fd2a0 tx=0x7f6efc00b040 comp rx=0 tx=0).stop 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 msgr2=0x7f6f0c0fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 0x7f6f0c0fb990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6f00000c00 tx=0x7f6f00009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 shutdown_connections 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f6ef40692a0 0x7f6ef406b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6f0c0fda80 0x7f6f0c0fd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 --2- 172.21.15.77:0/2473359596 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6f0c1003a0 0x7f6f0c0fb990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:10.941 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 >> 172.21.15.77:0/2473359596 conn(0x7f6f0c0f74e0 msgr2=0x7f6f0c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 shutdown_connections 2024-02-19T19:00:10.942 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:10.922+0000 7f6f11d93700 1 -- 172.21.15.77:0/2473359596 wait complete. 2024-02-19T19:00:10.951 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T19:00:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 -- 172.21.15.77:0/206274314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fcef0 msgr2=0x7f69fc0fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.044 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 --2- 172.21.15.77:0/206274314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fcef0 0x7f69fc0fd320 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009ce0 tx=0x7f69f4004d10 comp rx=0 tx=0).stop 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 -- 172.21.15.77:0/206274314 shutdown_connections 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 --2- 172.21.15.77:0/206274314 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69fc0fe0e0 0x7f69fc0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 --2- 172.21.15.77:0/206274314 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fcef0 0x7f69fc0fd320 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 -- 172.21.15.77:0/206274314 >> 172.21.15.77:0/206274314 conn(0x7f69fc0f8620 msgr2=0x7f69fc0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 -- 172.21.15.77:0/206274314 shutdown_connections 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 -- 172.21.15.77:0/206274314 wait complete. 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.025+0000 7f6a00d11700 1 Processor -- start 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f6a00d11700 1 -- start start 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f6a00d11700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69fc0fcef0 0x7f69fc10db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f6a00d11700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fe0e0 0x7f69fc10e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f6a00d11700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc10e670 con 0x7f69fc0fe0e0 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f6a00d11700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc10ef10 con 0x7f69fc0fcef0 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f69f9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fe0e0 0x7f69fc10e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.045 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f69fa59c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69fc0fcef0 0x7f69fc10db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f69f9d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fe0e0 0x7f69fc10e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38186/0 (socket says 172.21.15.77:38186) 2024-02-19T19:00:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f69f9d9b700 1 -- 172.21.15.77:0/4202405431 learned_addr learned my addr 172.21.15.77:0/4202405431 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f69fa59c700 1 -- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fe0e0 msgr2=0x7f69fc10e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f69fa59c700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fe0e0 0x7f69fc10e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.046 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f69fa59c700 1 -- 172.21.15.77:0/4202405431 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f4009940 con 0x7f69fc0fcef0 2024-02-19T19:00:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.026+0000 7f69fa59c700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69fc0fcef0 0x7f69fc10db30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f69f4015040 tx=0x7f69f4004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.027+0000 7f69eaffd700 1 -- 172.21.15.77:0/4202405431 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69f400e040 con 0x7f69fc0fcef0 2024-02-19T19:00:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.027+0000 7f69eaffd700 1 -- 172.21.15.77:0/4202405431 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f69f4008e20 con 0x7f69fc0fcef0 2024-02-19T19:00:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.027+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69fc10f190 con 0x7f69fc0fcef0 2024-02-19T19:00:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.027+0000 7f69eaffd700 1 -- 172.21.15.77:0/4202405431 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f69f401c620 con 0x7f69fc0fcef0 2024-02-19T19:00:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.027+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc10f6a0 con 0x7f69fc0fcef0 2024-02-19T19:00:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.028+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69fc0fc2a0 con 0x7f69fc0fcef0 2024-02-19T19:00:11.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.028+0000 7f69eaffd700 1 -- 172.21.15.77:0/4202405431 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f69f4012410 con 0x7f69fc0fcef0 2024-02-19T19:00:11.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.029+0000 7f69eaffd700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f69e40692b0 0x7f69e406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.029+0000 7f69f9d9b700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f69e40692b0 0x7f69e406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.029+0000 7f69eaffd700 1 -- 172.21.15.77:0/4202405431 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f69f4088c40 con 0x7f69fc0fcef0 2024-02-19T19:00:11.048 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.029+0000 7f69f9d9b700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f69e40692b0 0x7f69e406b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f69f00041d0 tx=0x7f69f000b040 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:11.056 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.037+0000 7f69eaffd700 1 -- 172.21.15.77:0/4202405431 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f69f4058960 con 0x7f69fc0fcef0 2024-02-19T19:00:11.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.241+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f69fc10fb40 con 0x7f69e40692b0 2024-02-19T19:00:11.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.243+0000 7f69eaffd700 1 -- 172.21.15.77:0/4202405431 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f69fc10fb40 con 0x7f69e40692b0 2024-02-19T19:00:11.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.244+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f69e40692b0 msgr2=0x7f69e406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.244+0000 7f6a00d11700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f69e40692b0 0x7f69e406b760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f69f00041d0 tx=0x7f69f000b040 comp rx=0 tx=0).stop 2024-02-19T19:00:11.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.244+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69fc0fcef0 msgr2=0x7f69fc10db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.244+0000 7f6a00d11700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69fc0fcef0 0x7f69fc10db30 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f69f4015040 tx=0x7f69f4004d10 comp rx=0 tx=0).stop 2024-02-19T19:00:11.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.245+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 shutdown_connections 2024-02-19T19:00:11.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.245+0000 7f6a00d11700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f69e40692b0 0x7f69e406b760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.245+0000 7f6a00d11700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f69fc0fcef0 0x7f69fc10db30 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.245+0000 7f6a00d11700 1 --2- 172.21.15.77:0/4202405431 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f69fc0fe0e0 0x7f69fc10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.245+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 >> 172.21.15.77:0/4202405431 conn(0x7f69fc0f8620 msgr2=0x7f69fc1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:11.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.245+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 shutdown_connections 2024-02-19T19:00:11.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.245+0000 7f6a00d11700 1 -- 172.21.15.77:0/4202405431 wait complete. 2024-02-19T19:00:11.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:11 smithi077 ceph-mon[29247]: pgmap v232: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:11.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 -- 172.21.15.77:0/607609200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a81003a0 msgr2=0x7f31a8102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 --2- 172.21.15.77:0/607609200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a81003a0 0x7f31a8102780 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3198004940 tx=0x7f3198009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:11.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 -- 172.21.15.77:0/607609200 shutdown_connections 2024-02-19T19:00:11.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 --2- 172.21.15.77:0/607609200 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a81003a0 0x7f31a8102780 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 --2- 172.21.15.77:0/607609200 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a80fda80 0x7f31a80ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 -- 172.21.15.77:0/607609200 >> 172.21.15.77:0/607609200 conn(0x7f31a80f74e0 msgr2=0x7f31a80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:11.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 -- 172.21.15.77:0/607609200 shutdown_connections 2024-02-19T19:00:11.366 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 -- 172.21.15.77:0/607609200 wait complete. 2024-02-19T19:00:11.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 Processor -- start 2024-02-19T19:00:11.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.347+0000 7f31af124700 1 -- start start 2024-02-19T19:00:11.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31af124700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a80fda80 0x7f31a80fd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.367 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31af124700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a81003a0 0x7f31a80fb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31af124700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80fbf90 con 0x7f31a80fda80 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31af124700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31a80fc100 con 0x7f31a81003a0 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31acec0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a80fda80 0x7f31a80fd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31acec0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a80fda80 0x7f31a80fd510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38208/0 (socket says 172.21.15.77:38208) 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31acec0700 1 -- 172.21.15.77:0/2113648167 learned_addr learned my addr 172.21.15.77:0/2113648167 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f319ffff700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a81003a0 0x7f31a80fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31acec0700 1 -- 172.21.15.77:0/2113648167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a81003a0 msgr2=0x7f31a80fb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31acec0700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a81003a0 0x7f31a80fb990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31acec0700 1 -- 172.21.15.77:0/2113648167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3198005040 con 0x7f31a80fda80 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31acec0700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a80fda80 0x7f31a80fd510 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f31a400ec90 tx=0x7f31a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f319d7fa700 1 -- 172.21.15.77:0/2113648167 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31a4007b30 con 0x7f31a80fda80 2024-02-19T19:00:11.368 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31a80fc3e0 con 0x7f31a80fda80 2024-02-19T19:00:11.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f319d7fa700 1 -- 172.21.15.77:0/2113648167 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f31a400f040 con 0x7f31a80fda80 2024-02-19T19:00:11.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f319d7fa700 1 -- 172.21.15.77:0/2113648167 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f31a400cc80 con 0x7f31a80fda80 2024-02-19T19:00:11.369 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.348+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31a8115910 con 0x7f31a80fda80 2024-02-19T19:00:11.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.350+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31a8107a50 con 0x7f31a80fda80 2024-02-19T19:00:11.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.350+0000 7f319d7fa700 1 -- 172.21.15.77:0/2113648167 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f31a4007c90 con 0x7f31a80fda80 2024-02-19T19:00:11.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.350+0000 7f319d7fa700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31900692a0 0x7f319006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.350+0000 7f319d7fa700 1 -- 172.21.15.77:0/2113648167 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f31a40875e0 con 0x7f31a80fda80 2024-02-19T19:00:11.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.350+0000 7f319ffff700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31900692a0 0x7f319006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.370 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.351+0000 7f319ffff700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31900692a0 0x7f319006b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3198009b30 tx=0x7f3198009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:11.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.355+0000 7f319d7fa700 1 -- 172.21.15.77:0/2113648167 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f31a4057320 con 0x7f31a80fda80 2024-02-19T19:00:11.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:11 smithi104 ceph-mon[42320]: pgmap v232: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:11.567 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.548+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f31a802b3b0 con 0x7f31900692a0 2024-02-19T19:00:11.576 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.558+0000 7f319d7fa700 1 -- 172.21.15.77:0/2113648167 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f31a802b3b0 con 0x7f31900692a0 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (5m) 83s ago 8m 26.7M - 0.25.0 c8568f914cd2 1e676c9ed794 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (9m) 83s ago 9m 6656k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (6m) 88s ago 6m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (5m) 83s ago 7m 86.8M - 9.4.7 954c08fa6188 2cb428963663 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (102s) 83s ago 102s 12.2M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (108s) 83s ago 107s 17.1M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (105s) 88s ago 105s 12.0M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (100s) 88s ago 100s 11.1M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:9283 running (10m) 83s ago 10m 480M - 17.2.7-521-gc80e7168 10b0c032c913 78483a70884c 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283 running (6m) 88s ago 6m 435M - 17.2.7-521-gc80e7168 10b0c032c913 5693aa48a572 2024-02-19T19:00:11.577 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (10m) 83s ago 10m 46.6M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (6m) 88s ago 6m 35.5M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (9m) 83s ago 9m 14.2M - 1.5.0 0da6a335fe13 18fa31ab4762 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (6m) 88s ago 6m 15.5M - 1.5.0 0da6a335fe13 5ec125749863 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (5m) 83s ago 5m 46.4M 4288M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (5m) 83s ago 5m 44.9M 4288M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (4m) 83s ago 4m 43.9M 4288M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (4m) 88s ago 4m 44.5M 5312M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (3m) 88s ago 3m 44.1M 5312M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (3m) 88s ago 3m 44.0M 5312M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T19:00:11.578 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (5m) 83s ago 6m 55.1M - 2.43.0 a07b618ecd1d 6dc30fbe4b32 2024-02-19T19:00:11.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31900692a0 msgr2=0x7f319006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31900692a0 0x7f319006b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3198009b30 tx=0x7f3198009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:11.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a80fda80 msgr2=0x7f31a80fd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a80fda80 0x7f31a80fd510 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f31a400ec90 tx=0x7f31a400b040 comp rx=0 tx=0).stop 2024-02-19T19:00:11.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 shutdown_connections 2024-02-19T19:00:11.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f31900692a0 0x7f319006b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f31a80fda80 0x7f31a80fd510 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 --2- 172.21.15.77:0/2113648167 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f31a81003a0 0x7f31a80fb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 >> 172.21.15.77:0/2113648167 conn(0x7f31a80f74e0 msgr2=0x7f31a80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:11.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.560+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 shutdown_connections 2024-02-19T19:00:11.580 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.561+0000 7f31af124700 1 -- 172.21.15.77:0/2113648167 wait complete. 2024-02-19T19:00:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.661+0000 7f58e6afb700 1 -- 172.21.15.77:0/2611946072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58e00fb220 msgr2=0x7f58e00fb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.661+0000 7f58e6afb700 1 --2- 172.21.15.77:0/2611946072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58e00fb220 0x7f58e00fb630 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f58d4004940 tx=0x7f58d4009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.661+0000 7f58e6afb700 1 -- 172.21.15.77:0/2611946072 shutdown_connections 2024-02-19T19:00:11.680 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.661+0000 7f58e6afb700 1 --2- 172.21.15.77:0/2611946072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58e00fe2b0 0x7f58e00fe700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.661+0000 7f58e6afb700 1 --2- 172.21.15.77:0/2611946072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58e00fb220 0x7f58e00fb630 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.661+0000 7f58e6afb700 1 -- 172.21.15.77:0/2611946072 >> 172.21.15.77:0/2611946072 conn(0x7f58e00f6ca0 msgr2=0x7f58e00f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.661+0000 7f58e6afb700 1 -- 172.21.15.77:0/2611946072 shutdown_connections 2024-02-19T19:00:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.661+0000 7f58e6afb700 1 -- 172.21.15.77:0/2611946072 wait complete. 2024-02-19T19:00:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58e6afb700 1 Processor -- start 2024-02-19T19:00:11.681 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58e6afb700 1 -- start start 2024-02-19T19:00:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58e6afb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58e00fb220 0x7f58e010d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58e6afb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58e00fe2b0 0x7f58e010dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58e6afb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e010e4e0 con 0x7f58e00fb220 2024-02-19T19:00:11.682 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58e6afb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58e010ed80 con 0x7f58e00fe2b0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58dffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58e00fe2b0 0x7f58e010dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58e4897700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58e00fb220 0x7f58e010d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58dffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58e00fe2b0 0x7f58e010dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.77:38908/0 (socket says 172.21.15.77:38908) 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.662+0000 7f58dffff700 1 -- 172.21.15.77:0/3083969729 learned_addr learned my addr 172.21.15.77:0/3083969729 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58dffff700 1 -- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58e00fb220 msgr2=0x7f58e010d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58dffff700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58e00fb220 0x7f58e010d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58dffff700 1 -- 172.21.15.77:0/3083969729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d4005040 con 0x7f58e00fe2b0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58dffff700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58e00fe2b0 0x7f58e010dee0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00eb40 tx=0x7f58cc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58dd7fa700 1 -- 172.21.15.77:0/3083969729 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58cc007cc0 con 0x7f58e00fe2b0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e010f000 con 0x7f58e00fe2b0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58dd7fa700 1 -- 172.21.15.77:0/3083969729 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58cc00f040 con 0x7f58e00fe2b0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58dd7fa700 1 -- 172.21.15.77:0/3083969729 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58cc00ca10 con 0x7f58e00fe2b0 2024-02-19T19:00:11.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.663+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e010f540 con 0x7f58e00fe2b0 2024-02-19T19:00:11.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.664+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e004bed0 con 0x7f58e00fe2b0 2024-02-19T19:00:11.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.665+0000 7f58dd7fa700 1 -- 172.21.15.77:0/3083969729 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7f58cc00cc80 con 0x7f58e00fe2b0 2024-02-19T19:00:11.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.665+0000 7f58dd7fa700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f58d00692a0 0x7f58d006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:11.684 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.665+0000 7f58dd7fa700 1 -- 172.21.15.77:0/3083969729 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f58cc086d80 con 0x7f58e00fe2b0 2024-02-19T19:00:11.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.665+0000 7f58e4897700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f58d00692a0 0x7f58d006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:11.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.666+0000 7f58e4897700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f58d00692a0 0x7f58d006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009b30 tx=0x7f58d4009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:11.688 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.669+0000 7f58dd7fa700 1 -- 172.21.15.77:0/3083969729 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7f58cc04dbd0 con 0x7f58e00fe2b0 2024-02-19T19:00:11.954 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.936+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f58e0105a40 con 0x7f58e00fe2b0 2024-02-19T19:00:11.955 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.936+0000 7f58dd7fa700 1 -- 172.21.15.77:0/3083969729 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f58cc056b70 con 0x7f58e00fe2b0 2024-02-19T19:00:11.955 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:00:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T19:00:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:00:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:00:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T19:00:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:00:11.955 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 14 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T19:00:11.956 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:00:11.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f58d00692a0 msgr2=0x7f58d006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f58d00692a0 0x7f58d006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58d4009b30 tx=0x7f58d4009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:11.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58e00fe2b0 msgr2=0x7f58e010dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:11.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58e00fe2b0 0x7f58e010dee0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f58cc00eb40 tx=0x7f58cc00b040 comp rx=0 tx=0).stop 2024-02-19T19:00:11.957 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 shutdown_connections 2024-02-19T19:00:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7f58d00692a0 0x7f58d006b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58e00fb220 0x7f58e010d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 --2- 172.21.15.77:0/3083969729 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58e00fe2b0 0x7f58e010dee0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 >> 172.21.15.77:0/3083969729 conn(0x7f58e00f6ca0 msgr2=0x7f58e01014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 shutdown_connections 2024-02-19T19:00:11.958 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:11.939+0000 7f58e6afb700 1 -- 172.21.15.77:0/3083969729 wait complete. 2024-02-19T19:00:12.059 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/3778260559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fcfb0 msgr2=0x7fe8e00fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 --2- 172.21.15.77:0/3778260559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fcfb0 0x7fe8e00fd420 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4004940 tx=0x7fe8d4009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/3778260559 shutdown_connections 2024-02-19T19:00:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 --2- 172.21.15.77:0/3778260559 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fcfb0 0x7fe8e00fd420 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 --2- 172.21.15.77:0/3778260559 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8e00fbe40 0x7fe8e00fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/3778260559 >> 172.21.15.77:0/3778260559 conn(0x7fe8e00f74e0 msgr2=0x7fe8e00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/3778260559 shutdown_connections 2024-02-19T19:00:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/3778260559 wait complete. 2024-02-19T19:00:12.060 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 Processor -- start 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.041+0000 7fe8e5d8c700 1 -- start start 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8e5d8c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fbe40 0x7fe8e010c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8e5d8c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8e00fcfb0 0x7fe8e010cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8e5d8c700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e010d310 con 0x7fe8e00fbe40 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8e5d8c700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e010dbb0 con 0x7fe8e00fcfb0 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8df7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fbe40 0x7fe8e010c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8df7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fbe40 0x7fe8e010c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38240/0 (socket says 172.21.15.77:38240) 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8df7fe700 1 -- 172.21.15.77:0/996278066 learned_addr learned my addr 172.21.15.77:0/996278066 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8deffd700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8e00fcfb0 0x7fe8e010cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8df7fe700 1 -- 172.21.15.77:0/996278066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8e00fcfb0 msgr2=0x7fe8e010cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8df7fe700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8e00fcfb0 0x7fe8e010cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.062 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8df7fe700 1 -- 172.21.15.77:0/996278066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8d4005040 con 0x7fe8e00fbe40 2024-02-19T19:00:12.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8df7fe700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fbe40 0x7fe8e010c7d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000eb40 tx=0x7fe8d000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:12.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8c7fff700 1 -- 172.21.15.77:0/996278066 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8d0007cc0 con 0x7fe8e00fbe40 2024-02-19T19:00:12.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e010de30 con 0x7fe8e00fbe40 2024-02-19T19:00:12.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8c7fff700 1 -- 172.21.15.77:0/996278066 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe8d000f040 con 0x7fe8e00fbe40 2024-02-19T19:00:12.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8c7fff700 1 -- 172.21.15.77:0/996278066 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe8d000ca10 con 0x7fe8e00fbe40 2024-02-19T19:00:12.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.042+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e010e370 con 0x7fe8e00fbe40 2024-02-19T19:00:12.063 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.043+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e004bed0 con 0x7fe8e00fbe40 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.044+0000 7fe8c7fff700 1 -- 172.21.15.77:0/996278066 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 19) v1 ==== 86621+0+0 (secure 0 0 0) 0x7fe8d000cc80 con 0x7fe8e00fbe40 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.044+0000 7fe8c7fff700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe8c80692a0 0x7fe8c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.044+0000 7fe8deffd700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe8c80692a0 0x7fe8c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.044+0000 7fe8c7fff700 1 -- 172.21.15.77:0/996278066 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe8d0086d80 con 0x7fe8e00fbe40 2024-02-19T19:00:12.064 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.045+0000 7fe8deffd700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe8c80692a0 0x7fe8c806b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4009b30 tx=0x7fe8d4009e30 comp rx=0 tx=0).ready entity=mgr.14270 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:12.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.049+0000 7fe8c7fff700 1 -- 172.21.15.77:0/996278066 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175031 (secure 0 0 0) 0x7fe8d004dbd0 con 0x7fe8e00fbe40 2024-02-19T19:00:12.270 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.251+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 --> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8e010e790 con 0x7fe8c80692a0 2024-02-19T19:00:12.271 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.252+0000 7fe8c7fff700 1 -- 172.21.15.77:0/996278066 <== mgr.14270 v2:172.21.15.77:6800/2462366592 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7fe8e010e790 con 0x7fe8c80692a0 2024-02-19T19:00:12.271 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:00:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:00:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T19:00:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T19:00:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-02-19T19:00:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "0/2 daemons upgraded", 2024-02-19T19:00:12.271 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526 image on host smithi104", 2024-02-19T19:00:12.272 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T19:00:12.272 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:00:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe8c80692a0 msgr2=0x7fe8c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe8c80692a0 0x7fe8c806b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe8d4009b30 tx=0x7fe8d4009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fbe40 msgr2=0x7fe8e010c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fbe40 0x7fe8e010c7d0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe8d000eb40 tx=0x7fe8d000b040 comp rx=0 tx=0).stop 2024-02-19T19:00:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 shutdown_connections 2024-02-19T19:00:12.273 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:6800/2462366592,v1:172.21.15.77:6801/2462366592] conn(0x7fe8c80692a0 0x7fe8c806b750 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8e00fbe40 0x7fe8e010c7d0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 --2- 172.21.15.77:0/996278066 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8e00fcfb0 0x7fe8e010cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.254+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 >> 172.21.15.77:0/996278066 conn(0x7fe8e00f74e0 msgr2=0x7fe8e0100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.255+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 shutdown_connections 2024-02-19T19:00:12.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:12.255+0000 7fe8e5d8c700 1 -- 172.21.15.77:0/996278066 wait complete. 2024-02-19T19:00:12.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:12 smithi077 ceph-mon[29247]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:12.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:12 smithi077 ceph-mon[29247]: from='client.24339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:12.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:12 smithi077 ceph-mon[29247]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:12.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:12 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3083969729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:00:12.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:12 smithi104 ceph-mon[42320]: from='client.14624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:12.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:12 smithi104 ceph-mon[42320]: from='client.24339 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:12.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:12 smithi104 ceph-mon[42320]: from='client.14632 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:12.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:12 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3083969729' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:00:13.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:13 smithi077 ceph-mon[29247]: pgmap v233: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:13.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:13 smithi077 ceph-mon[29247]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:13.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:13 smithi104 ceph-mon[42320]: pgmap v233: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:13.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:13 smithi104 ceph-mon[42320]: from='client.14640 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:15.544 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:15 smithi104 ceph-mon[42320]: pgmap v234: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:15.544 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:15 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:15.544 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:15 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.sdpzmi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:00:15.544 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:15 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:00:15.544 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:15 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:15.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:15 smithi077 ceph-mon[29247]: pgmap v234: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:15.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:15 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:15.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:15 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.sdpzmi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:00:15.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:15 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:00:15.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:15 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:16.457 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:16 smithi104 ceph-mon[42320]: Upgrade: Updating mgr.smithi104.sdpzmi 2024-02-19T19:00:16.457 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:16 smithi104 ceph-mon[42320]: Deploying daemon mgr.smithi104.sdpzmi on smithi104 2024-02-19T19:00:16.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:16 smithi077 ceph-mon[29247]: Upgrade: Updating mgr.smithi104.sdpzmi 2024-02-19T19:00:16.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:16 smithi077 ceph-mon[29247]: Deploying daemon mgr.smithi104.sdpzmi on smithi104 2024-02-19T19:00:17.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:17 smithi077 ceph-mon[29247]: pgmap v235: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:17.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:17 smithi104 ceph-mon[42320]: pgmap v235: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:19.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:18 smithi077 ceph-mon[29247]: pgmap v236: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:19.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:18 smithi104 ceph-mon[42320]: pgmap v236: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:20.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:19 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:20.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:19 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:21.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:20 smithi077 ceph-mon[29247]: pgmap v237: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:20 smithi104 ceph-mon[42320]: pgmap v237: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:23.525 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:23 smithi077 ceph-mon[29247]: pgmap v238: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:23.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:23 smithi104 ceph-mon[42320]: pgmap v238: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:25.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:25 smithi077 ceph-mon[29247]: pgmap v239: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:25.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:25 smithi104 ceph-mon[42320]: pgmap v239: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:27.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:26 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:26 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:26 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:26 smithi104 ceph-mon[42320]: pgmap v240: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:27.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:26 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:27.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:26 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:27.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:26 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:27.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:26 smithi077 ceph-mon[29247]: pgmap v240: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:29.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:28 smithi104 ceph-mon[42320]: pgmap v241: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:29.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:28 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:29.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:28 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:29.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:28 smithi077 ceph-mon[29247]: pgmap v241: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:29.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:28 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:29.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:28 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:30.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:30.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:30 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:30.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:30.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:30 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:31.660 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:31 smithi104 ceph-mon[42320]: pgmap v242: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:31.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:31 smithi077 ceph-mon[29247]: pgmap v242: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:32.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:32 smithi077 ceph-mon[29247]: Standby manager daemon smithi104.sdpzmi restarted 2024-02-19T19:00:32.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:32 smithi077 ceph-mon[29247]: Standby manager daemon smithi104.sdpzmi started 2024-02-19T19:00:32.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:32 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/crt"}]: dispatch 2024-02-19T19:00:32.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:32 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:00:32.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:32 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/key"}]: dispatch 2024-02-19T19:00:32.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:32 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:00:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:32 smithi104 ceph-mon[42320]: Standby manager daemon smithi104.sdpzmi restarted 2024-02-19T19:00:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:32 smithi104 ceph-mon[42320]: Standby manager daemon smithi104.sdpzmi started 2024-02-19T19:00:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:32 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/crt"}]: dispatch 2024-02-19T19:00:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:32 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:00:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:32 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/key"}]: dispatch 2024-02-19T19:00:32.582 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:32 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:00:33.637 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:33 smithi104 ceph-mon[42320]: pgmap v243: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:33.637 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:33 smithi104 ceph-mon[42320]: mgrmap e20: smithi077.yiiqpd(active, since 6m), standbys: smithi104.sdpzmi 2024-02-19T19:00:33.637 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:33.637 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:33.637 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:33.638 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:33.638 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:33 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:33.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:33 smithi077 ceph-mon[29247]: pgmap v243: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:33.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:33 smithi077 ceph-mon[29247]: mgrmap e20: smithi077.yiiqpd(active, since 6m), standbys: smithi104.sdpzmi 2024-02-19T19:00:33.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:33.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:33.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:33.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:33.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:33 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr fail", "who": "smithi077.yiiqpd"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: osdmap e47: 6 total, 6 up, 6 in 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "mgr fail", "who": "smithi077.yiiqpd"}]': finished 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: mgrmap e21: smithi104.sdpzmi(active, starting, since 0.00649611s) 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.ovassg"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.jmjalp"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.rudtoq"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.wpcqst"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr metadata", "who": "smithi104.sdpzmi", "id": "smithi104.sdpzmi"}]: dispatch 2024-02-19T19:00:34.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: Manager daemon smithi104.sdpzmi is now available 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.sdpzmi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.sdpzmi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:00:34.811 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.sdpzmi/trash_purge_schedule"}]: dispatch 2024-02-19T19:00:34.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.sdpzmi/trash_purge_schedule"}]: dispatch 2024-02-19T19:00:34.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:34.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:34.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: Deploying cephadm binary to smithi104 2024-02-19T19:00:34.812 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:34 smithi077 ceph-mon[29247]: Deploying cephadm binary to smithi077 2024-02-19T19:00:34.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:34.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr fail", "who": "smithi077.yiiqpd"}]: dispatch 2024-02-19T19:00:34.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: osdmap e47: 6 total, 6 up, 6 in 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.14270 172.21.15.77:0/3592212936' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "mgr fail", "who": "smithi077.yiiqpd"}]': finished 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: mgrmap e21: smithi104.sdpzmi(active, starting, since 0.00649611s) 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.ovassg"}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.jmjalp"}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.rudtoq"}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.wpcqst"}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr metadata", "who": "smithi104.sdpzmi", "id": "smithi104.sdpzmi"}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:00:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: Manager daemon smithi104.sdpzmi is now available 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.sdpzmi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.sdpzmi/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.sdpzmi/trash_purge_schedule"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi104.sdpzmi/trash_purge_schedule"}]: dispatch 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: Deploying cephadm binary to smithi104 2024-02-19T19:00:34.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:34 smithi104 ceph-mon[42320]: Deploying cephadm binary to smithi077 2024-02-19T19:00:35.675 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:35 smithi077 ceph-mon[29247]: [19/Feb/2024:19:00:34] ENGINE Bus STARTING 2024-02-19T19:00:35.675 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:35 smithi077 ceph-mon[29247]: [19/Feb/2024:19:00:34] ENGINE Serving on https://172.21.15.104:7150 2024-02-19T19:00:35.675 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:35 smithi077 ceph-mon[29247]: mgrmap e22: smithi104.sdpzmi(active, since 1.00744s) 2024-02-19T19:00:35.675 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:35 smithi077 ceph-mon[29247]: pgmap v3: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:35.675 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:35 smithi077 ceph-mon[29247]: [19/Feb/2024:19:00:34] ENGINE Serving on http://172.21.15.104:8765 2024-02-19T19:00:35.675 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:35 smithi077 ceph-mon[29247]: [19/Feb/2024:19:00:34] ENGINE Bus STARTED 2024-02-19T19:00:35.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:35 smithi104 ceph-mon[42320]: [19/Feb/2024:19:00:34] ENGINE Bus STARTING 2024-02-19T19:00:35.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:35 smithi104 ceph-mon[42320]: [19/Feb/2024:19:00:34] ENGINE Serving on https://172.21.15.104:7150 2024-02-19T19:00:35.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:35 smithi104 ceph-mon[42320]: mgrmap e22: smithi104.sdpzmi(active, since 1.00744s) 2024-02-19T19:00:35.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:35 smithi104 ceph-mon[42320]: pgmap v3: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:35.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:35 smithi104 ceph-mon[42320]: [19/Feb/2024:19:00:34] ENGINE Serving on http://172.21.15.104:8765 2024-02-19T19:00:35.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:35 smithi104 ceph-mon[42320]: [19/Feb/2024:19:00:34] ENGINE Bus STARTED 2024-02-19T19:00:36.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:36 smithi077 ceph-mon[29247]: pgmap v4: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:36.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:36 smithi104 ceph-mon[42320]: pgmap v4: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:37.809 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:37 smithi077 ceph-mon[29247]: mgrmap e23: smithi104.sdpzmi(active, since 3s) 2024-02-19T19:00:37.813 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:37 smithi104 ceph-mon[42320]: mgrmap e23: smithi104.sdpzmi(active, since 3s) 2024-02-19T19:00:39.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:38 smithi077 ceph-mon[29247]: pgmap v5: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:39.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:38 smithi104 ceph-mon[42320]: pgmap v5: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:40.243 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:40 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:40.243 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:40 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:40.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:40 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:40.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:40 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:41.459 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:41 smithi077 ceph-mon[29247]: pgmap v6: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:41.460 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:41 smithi077 ceph-mon[29247]: Standby manager daemon smithi077.yiiqpd started 2024-02-19T19:00:41.460 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:41 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.77:0/680861443' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.yiiqpd/crt"}]: dispatch 2024-02-19T19:00:41.460 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:41 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.77:0/680861443' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:00:41.460 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:41 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.77:0/680861443' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.yiiqpd/key"}]: dispatch 2024-02-19T19:00:41.460 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:41 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.77:0/680861443' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:00:41.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:41 smithi104 ceph-mon[42320]: pgmap v6: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:41.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:41 smithi104 ceph-mon[42320]: Standby manager daemon smithi077.yiiqpd started 2024-02-19T19:00:41.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:41 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.77:0/680861443' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.yiiqpd/crt"}]: dispatch 2024-02-19T19:00:41.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:41 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.77:0/680861443' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:00:41.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:41 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.77:0/680861443' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.yiiqpd/key"}]: dispatch 2024-02-19T19:00:41.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:41 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.77:0/680861443' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 -- 172.21.15.77:0/2072801125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fbe40 msgr2=0x7f62b80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 --2- 172.21.15.77:0/2072801125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fbe40 0x7f62b80fc250 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f62ac004940 tx=0x7f62ac009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 -- 172.21.15.77:0/2072801125 shutdown_connections 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 --2- 172.21.15.77:0/2072801125 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b80fcfb0 0x7f62b80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 --2- 172.21.15.77:0/2072801125 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fbe40 0x7f62b80fc250 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 -- 172.21.15.77:0/2072801125 >> 172.21.15.77:0/2072801125 conn(0x7f62b80f7520 msgr2=0x7f62b80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 -- 172.21.15.77:0/2072801125 shutdown_connections 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 -- 172.21.15.77:0/2072801125 wait complete. 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 Processor -- start 2024-02-19T19:00:42.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.365+0000 7f62beb1e700 1 -- start start 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62beb1e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b80fbe40 0x7f62b810d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62beb1e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fcfb0 0x7f62b810de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62beb1e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b810e400 con 0x7f62b80fcfb0 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62beb1e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b810eca0 con 0x7f62b80fbe40 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fcfb0 0x7f62b810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b7fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fcfb0 0x7f62b810de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57646/0 (socket says 172.21.15.77:57646) 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b7fff700 1 -- 172.21.15.77:0/790747315 learned_addr learned my addr 172.21.15.77:0/790747315 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62bc8ba700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b80fbe40 0x7f62b810d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b7fff700 1 -- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b80fbe40 msgr2=0x7f62b810d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b7fff700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b80fbe40 0x7f62b810d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b7fff700 1 -- 172.21.15.77:0/790747315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ac005040 con 0x7f62b80fcfb0 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b7fff700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fcfb0 0x7f62b810de00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f62a400eb40 tx=0x7f62a400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b57fa700 1 -- 172.21.15.77:0/790747315 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62a4007cc0 con 0x7f62b80fcfb0 2024-02-19T19:00:42.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b57fa700 1 -- 172.21.15.77:0/790747315 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62a400f040 con 0x7f62b80fcfb0 2024-02-19T19:00:42.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b810ef20 con 0x7f62b80fcfb0 2024-02-19T19:00:42.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62b57fa700 1 -- 172.21.15.77:0/790747315 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62a400ca10 con 0x7f62b80fcfb0 2024-02-19T19:00:42.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.366+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b810f460 con 0x7f62b80fcfb0 2024-02-19T19:00:42.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.368+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62b8107980 con 0x7f62b80fcfb0 2024-02-19T19:00:42.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.368+0000 7f62b57fa700 1 -- 172.21.15.77:0/790747315 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7f62a400cc80 con 0x7f62b80fcfb0 2024-02-19T19:00:42.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.368+0000 7f62b57fa700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f62a806b2e0 0x7f62a806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.368+0000 7f62b57fa700 1 -- 172.21.15.77:0/790747315 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f62a40899b0 con 0x7f62b80fcfb0 2024-02-19T19:00:42.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.368+0000 7f62bc8ba700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f62a806b2e0 0x7f62a806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.369+0000 7f62bc8ba700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f62a806b2e0 0x7f62a806d790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009b30 tx=0x7f62ac009e30 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:42.391 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.373+0000 7f62b57fa700 1 -- 172.21.15.77:0/790747315 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f62a404f620 con 0x7f62b80fcfb0 2024-02-19T19:00:42.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:42 smithi077 ceph-mon[29247]: mgrmap e24: smithi104.sdpzmi(active, since 7s), standbys: smithi077.yiiqpd 2024-02-19T19:00:42.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:42 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr metadata", "who": "smithi077.yiiqpd", "id": "smithi077.yiiqpd"}]: dispatch 2024-02-19T19:00:42.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:42 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:42.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:42 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:42.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:42 smithi104 ceph-mon[42320]: mgrmap e24: smithi104.sdpzmi(active, since 7s), standbys: smithi077.yiiqpd 2024-02-19T19:00:42.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:42 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr metadata", "who": "smithi077.yiiqpd", "id": "smithi077.yiiqpd"}]: dispatch 2024-02-19T19:00:42.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:42 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:42.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:42 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:42.609 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.591+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f62b802b310 con 0x7f62a806b2e0 2024-02-19T19:00:42.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.592+0000 7f62b57fa700 1 -- 172.21.15.77:0/790747315 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f62b802b310 con 0x7f62a806b2e0 2024-02-19T19:00:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f62a806b2e0 msgr2=0x7f62a806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f62a806b2e0 0x7f62a806d790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009b30 tx=0x7f62ac009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:42.613 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fcfb0 msgr2=0x7f62b810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fcfb0 0x7f62b810de00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f62a400eb40 tx=0x7f62a400b040 comp rx=0 tx=0).stop 2024-02-19T19:00:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 shutdown_connections 2024-02-19T19:00:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f62a806b2e0 0x7f62a806d790 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f62b80fbe40 0x7f62b810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 --2- 172.21.15.77:0/790747315 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f62b80fcfb0 0x7f62b810de00 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 >> 172.21.15.77:0/790747315 conn(0x7f62b80f7520 msgr2=0x7f62b80f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 shutdown_connections 2024-02-19T19:00:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.594+0000 7f62beb1e700 1 -- 172.21.15.77:0/790747315 wait complete. 2024-02-19T19:00:42.624 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T19:00:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 -- 172.21.15.77:0/1912520130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fcef0 msgr2=0x7eff340fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 --2- 172.21.15.77:0/1912520130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fcef0 0x7eff340fd320 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7eff24004940 tx=0x7eff24009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 -- 172.21.15.77:0/1912520130 shutdown_connections 2024-02-19T19:00:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 --2- 172.21.15.77:0/1912520130 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff340fe0e0 0x7eff340fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 --2- 172.21.15.77:0/1912520130 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fcef0 0x7eff340fd320 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 -- 172.21.15.77:0/1912520130 >> 172.21.15.77:0/1912520130 conn(0x7eff340f8620 msgr2=0x7eff340faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 -- 172.21.15.77:0/1912520130 shutdown_connections 2024-02-19T19:00:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 -- 172.21.15.77:0/1912520130 wait complete. 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 Processor -- start 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 -- start start 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff340fcef0 0x7eff3410d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fe0e0 0x7eff3410de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3410e420 con 0x7eff340fe0e0 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.710+0000 7eff391aa700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3410ecc0 con 0x7eff340fcef0 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff3259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fe0e0 0x7eff3410de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff32d9d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff340fcef0 0x7eff3410d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff3259c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fe0e0 0x7eff3410de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57652/0 (socket says 172.21.15.77:57652) 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff3259c700 1 -- 172.21.15.77:0/1024529255 learned_addr learned my addr 172.21.15.77:0/1024529255 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff32d9d700 1 -- 172.21.15.77:0/1024529255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fe0e0 msgr2=0x7eff3410de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff32d9d700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fe0e0 0x7eff3410de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff32d9d700 1 -- 172.21.15.77:0/1024529255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff24005040 con 0x7eff340fcef0 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff3259c700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fe0e0 0x7eff3410de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff32d9d700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff340fcef0 0x7eff3410d8e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff2400b0d0 tx=0x7eff24009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff1b7fe700 1 -- 172.21.15.77:0/1024529255 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff2400e070 con 0x7eff340fcef0 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff1b7fe700 1 -- 172.21.15.77:0/1024529255 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7eff2401c400 con 0x7eff340fcef0 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3410ef40 con 0x7eff340fcef0 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff1b7fe700 1 -- 172.21.15.77:0/1024529255 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff2400b670 con 0x7eff340fcef0 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.711+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3410f450 con 0x7eff340fcef0 2024-02-19T19:00:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.712+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3404bed0 con 0x7eff340fcef0 2024-02-19T19:00:42.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.713+0000 7eff1b7fe700 1 -- 172.21.15.77:0/1024529255 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7eff2401ca20 con 0x7eff340fcef0 2024-02-19T19:00:42.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.713+0000 7eff1b7fe700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7eff1c06b2e0 0x7eff1c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:42.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.713+0000 7eff1b7fe700 1 -- 172.21.15.77:0/1024529255 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7eff2408aff0 con 0x7eff340fcef0 2024-02-19T19:00:42.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.713+0000 7eff3259c700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7eff1c06b2e0 0x7eff1c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:42.734 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.714+0000 7eff3259c700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7eff1c06b2e0 0x7eff1c06d790 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7eff280041d0 tx=0x7eff2800b040 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.718+0000 7eff1b7fe700 1 -- 172.21.15.77:0/1024529255 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7eff24058be0 con 0x7eff340fcef0 2024-02-19T19:00:42.959 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.941+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7eff3410f8a0 con 0x7eff1c06b2e0 2024-02-19T19:00:42.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.942+0000 7eff1b7fe700 1 -- 172.21.15.77:0/1024529255 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7eff3410f8a0 con 0x7eff1c06b2e0 2024-02-19T19:00:42.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.944+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7eff1c06b2e0 msgr2=0x7eff1c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.944+0000 7eff391aa700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7eff1c06b2e0 0x7eff1c06d790 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7eff280041d0 tx=0x7eff2800b040 comp rx=0 tx=0).stop 2024-02-19T19:00:42.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.944+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff340fcef0 msgr2=0x7eff3410d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:42.963 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.944+0000 7eff391aa700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff340fcef0 0x7eff3410d8e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7eff2400b0d0 tx=0x7eff24009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.944+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 shutdown_connections 2024-02-19T19:00:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.944+0000 7eff391aa700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7eff1c06b2e0 0x7eff1c06d790 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.944+0000 7eff391aa700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7eff340fcef0 0x7eff3410d8e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.944+0000 7eff391aa700 1 --2- 172.21.15.77:0/1024529255 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7eff340fe0e0 0x7eff3410de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.945+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 >> 172.21.15.77:0/1024529255 conn(0x7eff340f8620 msgr2=0x7eff341013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.945+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 shutdown_connections 2024-02-19T19:00:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:42.945+0000 7eff391aa700 1 -- 172.21.15.77:0/1024529255 wait complete. 2024-02-19T19:00:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.047+0000 7f1e508c2700 1 -- 172.21.15.77:0/1374178791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 msgr2=0x7f1e4c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.047+0000 7f1e508c2700 1 --2- 172.21.15.77:0/1374178791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 0x7f1e4c0fe550 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f1e40004940 tx=0x7f1e40009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.047+0000 7f1e508c2700 1 -- 172.21.15.77:0/1374178791 shutdown_connections 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.047+0000 7f1e508c2700 1 --2- 172.21.15.77:0/1374178791 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 0x7f1e4c0fe550 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.047+0000 7f1e508c2700 1 --2- 172.21.15.77:0/1374178791 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e4c0fcef0 0x7f1e4c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.047+0000 7f1e508c2700 1 -- 172.21.15.77:0/1374178791 >> 172.21.15.77:0/1374178791 conn(0x7f1e4c0f8620 msgr2=0x7f1e4c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e508c2700 1 -- 172.21.15.77:0/1374178791 shutdown_connections 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e508c2700 1 -- 172.21.15.77:0/1374178791 wait complete. 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e508c2700 1 Processor -- start 2024-02-19T19:00:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e508c2700 1 -- start start 2024-02-19T19:00:43.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e508c2700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e4c0fcef0 0x7f1e4c071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.068 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e508c2700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 0x7f1e4c0702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e508c2700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c0708e0 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e508c2700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4c070a50 con 0x7f1e4c0fcef0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.048+0000 7f1e49d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 0x7f1e4c0702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e49d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 0x7f1e4c0702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57670/0 (socket says 172.21.15.77:57670) 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e49d9b700 1 -- 172.21.15.77:0/3422024418 learned_addr learned my addr 172.21.15.77:0/3422024418 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e4a59c700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e4c0fcef0 0x7f1e4c071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e49d9b700 1 -- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e4c0fcef0 msgr2=0x7f1e4c071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e49d9b700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e4c0fcef0 0x7f1e4c071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e49d9b700 1 -- 172.21.15.77:0/3422024418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e40005040 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e49d9b700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 0x7f1e4c0702e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1e400093b0 tx=0x7f1e40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e3affd700 1 -- 172.21.15.77:0/3422024418 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e4000e040 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e3affd700 1 -- 172.21.15.77:0/3422024418 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1e4000bcc0 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4c070cd0 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e3affd700 1 -- 172.21.15.77:0/3422024418 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e4001c630 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.049+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4c115900 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.050+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e4c04bed0 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.051+0000 7f1e3affd700 1 -- 172.21.15.77:0/3422024418 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7f1e40013980 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.051+0000 7f1e3affd700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f1e3406b2e0 0x7f1e3406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.051+0000 7f1e3affd700 1 -- 172.21.15.77:0/3422024418 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f1e4008b060 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.051+0000 7f1e4a59c700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f1e3406b2e0 0x7f1e3406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.052+0000 7f1e4a59c700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f1e3406b2e0 0x7f1e3406d790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c071bf0 tx=0x7f1e3c00b040 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.056+0000 7f1e3affd700 1 -- 172.21.15.77:0/3422024418 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f1e40058c90 con 0x7f1e4c0fe0e0 2024-02-19T19:00:43.195 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:43 smithi077 ceph-mon[29247]: pgmap v7: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-19T19:00:43.195 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:43 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:43.195 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:43 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:43.282 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.263+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e4c102ac0 con 0x7f1e3406b2e0 2024-02-19T19:00:43.292 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.274+0000 7f1e3affd700 1 -- 172.21.15.77:0/3422024418 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7f1e4c102ac0 con 0x7f1e3406b2e0 2024-02-19T19:00:43.292 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:00:43.292 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (6m) 115s ago 9m 26.7M - 0.25.0 c8568f914cd2 1e676c9ed794 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (9m) 115s ago 9m 6656k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (6m) 1s ago 6m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (6m) 115s ago 7m 86.8M - 9.4.7 954c08fa6188 2cb428963663 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (2m) 115s ago 2m 12.2M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (2m) 115s ago 2m 17.1M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (2m) 1s ago 2m 13.1M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (2m) 1s ago 2m 12.2M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:9283 running (10m) 115s ago 10m 480M - 17.2.7-521-gc80e7168 10b0c032c913 78483a70884c 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283 running (21s) 1s ago 6m 488M - 19.0.0-1450-gf78a58c0 f081f847070a bc512497a554 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (10m) 115s ago 11m 46.6M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (6m) 1s ago 6m 38.2M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (9m) 115s ago 9m 14.2M - 1.5.0 0da6a335fe13 18fa31ab4762 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (6m) 1s ago 6m 15.2M - 1.5.0 0da6a335fe13 5ec125749863 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (6m) 115s ago 6m 46.4M 4288M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (5m) 115s ago 5m 44.9M 4288M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (5m) 115s ago 5m 43.9M 4288M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T19:00:43.293 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (4m) 1s ago 4m 45.9M 5312M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T19:00:43.294 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (4m) 1s ago 4m 45.9M 5312M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T19:00:43.294 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (3m) 1s ago 3m 46.0M 5312M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T19:00:43.294 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (6m) 115s ago 7m 55.1M - 2.43.0 a07b618ecd1d 6dc30fbe4b32 2024-02-19T19:00:43.294 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.275+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f1e3406b2e0 msgr2=0x7f1e3406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.275+0000 7f1e508c2700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f1e3406b2e0 0x7f1e3406d790 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c071bf0 tx=0x7f1e3c00b040 comp rx=0 tx=0).stop 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.275+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 msgr2=0x7f1e4c0702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.275+0000 7f1e508c2700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 0x7f1e4c0702e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1e400093b0 tx=0x7f1e40009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.276+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 shutdown_connections 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.276+0000 7f1e508c2700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f1e3406b2e0 0x7f1e3406d790 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.276+0000 7f1e508c2700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f1e4c0fcef0 0x7f1e4c071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.276+0000 7f1e508c2700 1 --2- 172.21.15.77:0/3422024418 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f1e4c0fe0e0 0x7f1e4c0702e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.276+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 >> 172.21.15.77:0/3422024418 conn(0x7f1e4c0f8620 msgr2=0x7f1e4c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.276+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 shutdown_connections 2024-02-19T19:00:43.295 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.276+0000 7f1e508c2700 1 -- 172.21.15.77:0/3422024418 wait complete. 2024-02-19T19:00:43.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.376+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/2422045990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 msgr2=0x7f0cf40fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.376+0000 7f0cfc2a5700 1 --2- 172.21.15.77:0/2422045990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 0x7f0cf40fc250 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0004940 tx=0x7f0cf0009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.376+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/2422045990 shutdown_connections 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.376+0000 7f0cfc2a5700 1 --2- 172.21.15.77:0/2422045990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cf40fcfb0 0x7f0cf40fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.376+0000 7f0cfc2a5700 1 --2- 172.21.15.77:0/2422045990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 0x7f0cf40fc250 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.376+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/2422045990 >> 172.21.15.77:0/2422045990 conn(0x7f0cf40f74e0 msgr2=0x7f0cf40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.376+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/2422045990 shutdown_connections 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.376+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/2422045990 wait complete. 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfc2a5700 1 Processor -- start 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfc2a5700 1 -- start start 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfc2a5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 0x7f0cf4110690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfc2a5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cf40fcfb0 0x7f0cf4112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfc2a5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf4066480 con 0x7f0cf40fbe40 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfc2a5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf40665c0 con 0x7f0cf40fcfb0 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfa041700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 0x7f0cf4110690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfa041700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 0x7f0cf4110690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57686/0 (socket says 172.21.15.77:57686) 2024-02-19T19:00:43.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.377+0000 7f0cfa041700 1 -- 172.21.15.77:0/4264932331 learned_addr learned my addr 172.21.15.77:0/4264932331 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0cf9840700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cf40fcfb0 0x7f0cf4112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0cfa041700 1 -- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cf40fcfb0 msgr2=0x7f0cf4112be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0cfa041700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cf40fcfb0 0x7f0cf4112be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0cfa041700 1 -- 172.21.15.77:0/4264932331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0005040 con 0x7f0cf40fbe40 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0cfa041700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 0x7f0cf4110690 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00093b0 tx=0x7f0cf0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0ceaffd700 1 -- 172.21.15.77:0/4264932331 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cf000e040 con 0x7f0cf40fbe40 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0ceaffd700 1 -- 172.21.15.77:0/4264932331 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0cf000bcc0 con 0x7f0cf40fbe40 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf41131e0 con 0x7f0cf40fbe40 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0ceaffd700 1 -- 172.21.15.77:0/4264932331 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cf001c630 con 0x7f0cf40fbe40 2024-02-19T19:00:43.399 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.378+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf41136d0 con 0x7f0cf40fbe40 2024-02-19T19:00:43.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.379+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf404bed0 con 0x7f0cf40fbe40 2024-02-19T19:00:43.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.380+0000 7f0ceaffd700 1 -- 172.21.15.77:0/4264932331 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7f0cf0013980 con 0x7f0cf40fbe40 2024-02-19T19:00:43.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.380+0000 7f0ceaffd700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f0ce006b2e0 0x7f0ce006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.380+0000 7f0ceaffd700 1 -- 172.21.15.77:0/4264932331 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f0cf008b170 con 0x7f0cf40fbe40 2024-02-19T19:00:43.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.380+0000 7f0cf9840700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f0ce006b2e0 0x7f0ce006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.401 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.381+0000 7f0cf9840700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f0ce006b2e0 0x7f0ce006d790 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4004f60 tx=0x7f0ce400b040 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:43.403 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.385+0000 7f0ceaffd700 1 -- 172.21.15.77:0/4264932331 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f0cf00551d0 con 0x7f0cf40fbe40 2024-02-19T19:00:43.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:43 smithi104 ceph-mon[42320]: pgmap v7: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-19T19:00:43.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:43 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:43.467 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:43 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:43.716 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.697+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f0cf4113af0 con 0x7f0cf40fbe40 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.698+0000 7f0ceaffd700 1 -- 172.21.15.77:0/4264932331 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f0cf0058890 con 0x7f0cf40fbe40 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 1, 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T19:00:43.717 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:00:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T19:00:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T19:00:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:00:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T19:00:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 13, 2024-02-19T19:00:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:00:43.718 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T19:00:43.718 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:00:43.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.700+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f0ce006b2e0 msgr2=0x7f0ce006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.700+0000 7f0cfc2a5700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f0ce006b2e0 0x7f0ce006d790 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4004f60 tx=0x7f0ce400b040 comp rx=0 tx=0).stop 2024-02-19T19:00:43.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.700+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 msgr2=0x7f0cf4110690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.719 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.700+0000 7f0cfc2a5700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 0x7f0cf4110690 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00093b0 tx=0x7f0cf0009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:43.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.700+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 shutdown_connections 2024-02-19T19:00:43.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.701+0000 7f0cfc2a5700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f0ce006b2e0 0x7f0ce006d790 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.701+0000 7f0cfc2a5700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f0cf40fbe40 0x7f0cf4110690 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.701+0000 7f0cfc2a5700 1 --2- 172.21.15.77:0/4264932331 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f0cf40fcfb0 0x7f0cf4112be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.701+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 >> 172.21.15.77:0/4264932331 conn(0x7f0cf40f74e0 msgr2=0x7f0cf4100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:43.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.701+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 shutdown_connections 2024-02-19T19:00:43.720 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.701+0000 7f0cfc2a5700 1 -- 172.21.15.77:0/4264932331 wait complete. 2024-02-19T19:00:43.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.809+0000 7fefc14fd700 1 -- 172.21.15.77:0/3086190578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fbcc0 msgr2=0x7fefbc102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.828 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.809+0000 7fefc14fd700 1 --2- 172.21.15.77:0/3086190578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fbcc0 0x7fefbc102560 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fefb0004940 tx=0x7fefb0009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:43.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.810+0000 7fefc14fd700 1 -- 172.21.15.77:0/3086190578 shutdown_connections 2024-02-19T19:00:43.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.810+0000 7fefc14fd700 1 --2- 172.21.15.77:0/3086190578 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fbcc0 0x7fefbc102560 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.810+0000 7fefc14fd700 1 --2- 172.21.15.77:0/3086190578 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefbc0fb2e0 0x7fefbc0fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.810+0000 7fefc14fd700 1 -- 172.21.15.77:0/3086190578 >> 172.21.15.77:0/3086190578 conn(0x7fefbc0f6ca0 msgr2=0x7fefbc0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:43.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.810+0000 7fefc14fd700 1 -- 172.21.15.77:0/3086190578 shutdown_connections 2024-02-19T19:00:43.829 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.810+0000 7fefc14fd700 1 -- 172.21.15.77:0/3086190578 wait complete. 2024-02-19T19:00:43.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.810+0000 7fefc14fd700 1 Processor -- start 2024-02-19T19:00:43.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.810+0000 7fefc14fd700 1 -- start start 2024-02-19T19:00:43.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefc14fd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fb2e0 0x7fefbc0fde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefc14fd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefbc0fc2c0 0x7fefbc0fc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefc14fd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0fccc0 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.830 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefc14fd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0fce00 con 0x7fefbc0fc2c0 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefbaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fb2e0 0x7fefbc0fde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefbaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fb2e0 0x7fefbc0fde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:57698/0 (socket says 172.21.15.77:57698) 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefbaffd700 1 -- 172.21.15.77:0/1049931819 learned_addr learned my addr 172.21.15.77:0/1049931819 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefba7fc700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefbc0fc2c0 0x7fefbc0fc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefbaffd700 1 -- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefbc0fc2c0 msgr2=0x7fefbc0fc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefbaffd700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefbc0fc2c0 0x7fefbc0fc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefbaffd700 1 -- 172.21.15.77:0/1049931819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0005040 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefba7fc700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefbc0fc2c0 0x7fefbc0fc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefbaffd700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fb2e0 0x7fefbc0fde40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fefac002a50 tx=0x7fefac0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefa37fe700 1 -- 172.21.15.77:0/1049931819 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefac007cc0 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc0fd0e0 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.811+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc115da0 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.812+0000 7fefa37fe700 1 -- 172.21.15.77:0/1049931819 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefac00f040 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.812+0000 7fefa37fe700 1 -- 172.21.15.77:0/1049931819 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefac00da30 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.813+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefbc107a90 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.832 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.813+0000 7fefa37fe700 1 -- 172.21.15.77:0/1049931819 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7fefac00dc70 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.814+0000 7fefa37fe700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fefa406b2e0 0x7fefa406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:00:43.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.814+0000 7fefa37fe700 1 -- 172.21.15.77:0/1049931819 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fefac089aa0 con 0x7fefbc0fb2e0 2024-02-19T19:00:43.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.814+0000 7fefba7fc700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fefa406b2e0 0x7fefa406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:00:43.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.814+0000 7fefba7fc700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fefa406b2e0 0x7fefa406d790 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fefb000b3b0 tx=0x7fefb0009e30 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:00:43.841 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:43.823+0000 7fefa37fe700 1 -- 172.21.15.77:0/1049931819 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fefac057630 con 0x7fefbc0fb2e0 2024-02-19T19:00:44.069 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.050+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefbc0fbdd0 con 0x7fefa406b2e0 2024-02-19T19:00:44.070 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.051+0000 7fefa37fe700 1 -- 172.21.15.77:0/1049931819 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fefbc0fbdd0 con 0x7fefa406b2e0 2024-02-19T19:00:44.070 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:00:44.070 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:00:44.070 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T19:00:44.070 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T19:00:44.071 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-02-19T19:00:44.071 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "1/2 daemons upgraded", 2024-02-19T19:00:44.071 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-02-19T19:00:44.071 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T19:00:44.071 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fefa406b2e0 msgr2=0x7fefa406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fefa406b2e0 0x7fefa406d790 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fefb000b3b0 tx=0x7fefb0009e30 comp rx=0 tx=0).stop 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fb2e0 msgr2=0x7fefbc0fde40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fb2e0 0x7fefbc0fde40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fefac002a50 tx=0x7fefac0046b0 comp rx=0 tx=0).stop 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 shutdown_connections 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fefa406b2e0 0x7fefa406d790 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefbc0fb2e0 0x7fefbc0fde40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 --2- 172.21.15.77:0/1049931819 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefbc0fc2c0 0x7fefbc0fc730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 >> 172.21.15.77:0/1049931819 conn(0x7fefbc0f6ca0 msgr2=0x7fefbc0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 shutdown_connections 2024-02-19T19:00:44.073 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:00:44.054+0000 7fefc14fd700 1 -- 172.21.15.77:0/1049931819 wait complete. 2024-02-19T19:00:44.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:44 smithi104 ceph-mon[42320]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:44.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:44 smithi104 ceph-mon[42320]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:44.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:44 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:44.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:44 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:44.537 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:44 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/4264932331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:00:44.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:44 smithi077 ceph-mon[29247]: from='client.14674 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:44.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:44 smithi077 ceph-mon[29247]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:44.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:44 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:44.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:44 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:44.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:44 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/4264932331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:00:45.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:45.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: pgmap v8: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:00:45.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:45 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:45.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:45.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: pgmap v8: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:00:45.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='client.14690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:00:45.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:45.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:45.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:45 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:46.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:46 smithi077 ceph-mon[29247]: Adjusting osd_memory_target on smithi077 to 1557M 2024-02-19T19:00:46.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:46 smithi104 ceph-mon[42320]: Adjusting osd_memory_target on smithi077 to 1557M 2024-02-19T19:00:47.464 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:47 smithi104 ceph-mon[42320]: pgmap v9: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-19T19:00:47.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:47 smithi077 ceph-mon[29247]: pgmap v9: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-19T19:00:49.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: pgmap v10: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:00:49.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:49.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:49.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:49 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:49.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: pgmap v10: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:00:49.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:00:49.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:49 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:00:50.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: Adjusting osd_memory_target on smithi104 to 2581M 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: Updating smithi077:/etc/ceph/ceph.conf 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: Updating smithi104:/etc/ceph/ceph.conf 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: Adjusting osd_memory_target on smithi104 to 2581M 2024-02-19T19:00:50.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: Updating smithi077:/etc/ceph/ceph.conf 2024-02-19T19:00:50.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: Updating smithi104:/etc/ceph/ceph.conf 2024-02-19T19:00:50.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:50.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:51.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:51 smithi104 ceph-mon[42320]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:00:51.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:51 smithi104 ceph-mon[42320]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:00:51.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:51 smithi104 ceph-mon[42320]: pgmap v11: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:00:51.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:51 smithi104 ceph-mon[42320]: Reconfiguring node-exporter.smithi077 (dependencies changed)... 2024-02-19T19:00:51.206 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:51 smithi104 ceph-mon[42320]: Reconfiguring daemon node-exporter.smithi077 on smithi077 2024-02-19T19:00:51.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:51 smithi077 ceph-mon[29247]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:00:51.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:51 smithi077 ceph-mon[29247]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:00:51.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:51 smithi077 ceph-mon[29247]: pgmap v11: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:00:51.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:51 smithi077 ceph-mon[29247]: Reconfiguring node-exporter.smithi077 (dependencies changed)... 2024-02-19T19:00:51.493 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:51 smithi077 ceph-mon[29247]: Reconfiguring daemon node-exporter.smithi077 on smithi077 2024-02-19T19:00:53.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:53 smithi077 ceph-mon[29247]: pgmap v12: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:00:53.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:53 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:53.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:53 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:53.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:53 smithi077 ceph-mon[29247]: Reconfiguring alertmanager.smithi077 (dependencies changed)... 2024-02-19T19:00:53.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:53 smithi077 ceph-mon[29247]: Reconfiguring daemon alertmanager.smithi077 on smithi077 2024-02-19T19:00:53.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:53 smithi104 ceph-mon[42320]: pgmap v12: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:00:53.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:53 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:53.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:53 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:53.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:53 smithi104 ceph-mon[42320]: Reconfiguring alertmanager.smithi077 (dependencies changed)... 2024-02-19T19:00:53.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:53 smithi104 ceph-mon[42320]: Reconfiguring daemon alertmanager.smithi077 on smithi077 2024-02-19T19:00:55.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:55 smithi077 ceph-mon[29247]: pgmap v13: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:55.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:55 smithi104 ceph-mon[42320]: pgmap v13: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:57.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:57 smithi077 ceph-mon[29247]: pgmap v14: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:57.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:57 smithi104 ceph-mon[42320]: pgmap v14: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:59.441 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:59 smithi077 ceph-mon[29247]: pgmap v15: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:59.441 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:59 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:59.441 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:59 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:59.441 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:00:59 smithi077 ceph-mon[29247]: Reconfiguring grafana.smithi077 (dependencies changed)... 2024-02-19T19:00:59.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:59 smithi104 ceph-mon[42320]: pgmap v15: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:00:59.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:59 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:59.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:59 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:00:59.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:00:59 smithi104 ceph-mon[42320]: Reconfiguring grafana.smithi077 (dependencies changed)... 2024-02-19T19:01:00.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:00 smithi077 ceph-mon[29247]: Reconfiguring daemon grafana.smithi077 on smithi077 2024-02-19T19:01:00.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:00 smithi104 ceph-mon[42320]: Reconfiguring daemon grafana.smithi077 on smithi077 2024-02-19T19:01:01.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:01 smithi077 ceph-mon[29247]: pgmap v16: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:01.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:01 smithi104 ceph-mon[42320]: pgmap v16: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:03.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:03 smithi077 ceph-mon[29247]: pgmap v17: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:03.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:03 smithi104 ceph-mon[42320]: pgmap v17: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:04.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:04 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:04.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:04 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:05.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:05 smithi077 ceph-mon[29247]: pgmap v18: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:05.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:05 smithi104 ceph-mon[42320]: pgmap v18: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:07.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:07 smithi077 ceph-mon[29247]: pgmap v19: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:07.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:07 smithi104 ceph-mon[42320]: pgmap v19: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:09.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:09 smithi077 ceph-mon[29247]: pgmap v20: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:09.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:09 smithi104 ceph-mon[42320]: pgmap v20: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:11.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:11 smithi077 ceph-mon[29247]: pgmap v21: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:11.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:11 smithi104 ceph-mon[42320]: pgmap v21: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:12.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:12 smithi077 ceph-mon[29247]: pgmap v22: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:12.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:12 smithi104 ceph-mon[42320]: pgmap v22: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.168+0000 7f87d32ea700 1 -- 172.21.15.77:0/1931887386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc102430 msgr2=0x7f87cc104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.168+0000 7f87d32ea700 1 --2- 172.21.15.77:0/1931887386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc102430 0x7f87cc104810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f87bc004880 tx=0x7f87bc009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.168+0000 7f87d32ea700 1 -- 172.21.15.77:0/1931887386 shutdown_connections 2024-02-19T19:01:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.168+0000 7f87d32ea700 1 --2- 172.21.15.77:0/1931887386 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc102430 0x7f87cc104810 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.168+0000 7f87d32ea700 1 --2- 172.21.15.77:0/1931887386 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87cc066c70 0x7f87cc101ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.168+0000 7f87d32ea700 1 -- 172.21.15.77:0/1931887386 >> 172.21.15.77:0/1931887386 conn(0x7f87cc0f7500 msgr2=0x7f87cc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.168+0000 7f87d32ea700 1 -- 172.21.15.77:0/1931887386 shutdown_connections 2024-02-19T19:01:14.189 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.168+0000 7f87d32ea700 1 -- 172.21.15.77:0/1931887386 wait complete. 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d32ea700 1 Processor -- start 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d32ea700 1 -- start start 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d32ea700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87cc066c70 0x7f87cc10fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d32ea700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc102430 0x7f87cc1100b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d32ea700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc1106b0 con 0x7f87cc066c70 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d32ea700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87cc110f50 con 0x7f87cc102430 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d1086700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87cc066c70 0x7f87cc10fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d1086700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87cc066c70 0x7f87cc10fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50028/0 (socket says 172.21.15.77:50028) 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d1086700 1 -- 172.21.15.77:0/1079847062 learned_addr learned my addr 172.21.15.77:0/1079847062 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d0885700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc102430 0x7f87cc1100b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d1086700 1 -- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc102430 msgr2=0x7f87cc1100b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d1086700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc102430 0x7f87cc1100b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.169+0000 7f87d1086700 1 -- 172.21.15.77:0/1079847062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c8005040 con 0x7f87cc066c70 2024-02-19T19:01:14.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.170+0000 7f87d1086700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87cc066c70 0x7f87cc10fb70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f87c80096b0 tx=0x7f87c8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.170+0000 7f87c1ffb700 1 -- 172.21.15.77:0/1079847062 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87c8017070 con 0x7f87cc066c70 2024-02-19T19:01:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.170+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87bc005040 con 0x7f87cc066c70 2024-02-19T19:01:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.170+0000 7f87c1ffb700 1 -- 172.21.15.77:0/1079847062 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f87c800f040 con 0x7f87cc066c70 2024-02-19T19:01:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.170+0000 7f87c1ffb700 1 -- 172.21.15.77:0/1079847062 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f87c8013be0 con 0x7f87cc066c70 2024-02-19T19:01:14.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.170+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87cc111580 con 0x7f87cc066c70 2024-02-19T19:01:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.171+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87cc04bed0 con 0x7f87cc066c70 2024-02-19T19:01:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.171+0000 7f87c1ffb700 1 -- 172.21.15.77:0/1079847062 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7f87c800e840 con 0x7f87cc066c70 2024-02-19T19:01:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.172+0000 7f87c1ffb700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f87b806b2e0 0x7f87b806d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.172+0000 7f87c1ffb700 1 -- 172.21.15.77:0/1079847062 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f87c8089bf0 con 0x7f87cc066c70 2024-02-19T19:01:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.172+0000 7f87d0885700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f87b806b2e0 0x7f87b806d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.192 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.172+0000 7f87d0885700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f87b806b2e0 0x7f87b806d790 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0137c0 tx=0x7f87bc009e30 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:14.196 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.177+0000 7f87c1ffb700 1 -- 172.21.15.77:0/1079847062 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f87c8057700 con 0x7f87cc066c70 2024-02-19T19:01:14.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.421+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f87cc111bd0 con 0x7f87b806b2e0 2024-02-19T19:01:14.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.423+0000 7f87c1ffb700 1 -- 172.21.15.77:0/1079847062 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f87cc111bd0 con 0x7f87b806b2e0 2024-02-19T19:01:14.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f87b806b2e0 msgr2=0x7f87b806d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f87b806b2e0 0x7f87b806d790 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f87bc0137c0 tx=0x7f87bc009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:14.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87cc066c70 msgr2=0x7f87cc10fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87cc066c70 0x7f87cc10fb70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f87c80096b0 tx=0x7f87c8009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:14.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 shutdown_connections 2024-02-19T19:01:14.444 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f87b806b2e0 0x7f87b806d790 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f87cc066c70 0x7f87cc10fb70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 --2- 172.21.15.77:0/1079847062 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f87cc102430 0x7f87cc1100b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 >> 172.21.15.77:0/1079847062 conn(0x7f87cc0f7500 msgr2=0x7f87cc0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:14.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 shutdown_connections 2024-02-19T19:01:14.445 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.425+0000 7f87d32ea700 1 -- 172.21.15.77:0/1079847062 wait complete. 2024-02-19T19:01:14.455 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T19:01:14.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.536+0000 7fd9a3023700 1 -- 172.21.15.77:0/889827178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c1003a0 msgr2=0x7fd99c102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.555 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.536+0000 7fd9a3023700 1 --2- 172.21.15.77:0/889827178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c1003a0 0x7fd99c102780 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fd988004940 tx=0x7fd988009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.536+0000 7fd9a3023700 1 -- 172.21.15.77:0/889827178 shutdown_connections 2024-02-19T19:01:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.536+0000 7fd9a3023700 1 --2- 172.21.15.77:0/889827178 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c1003a0 0x7fd99c102780 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.536+0000 7fd9a3023700 1 --2- 172.21.15.77:0/889827178 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd99c0fda80 0x7fd99c0ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.536+0000 7fd9a3023700 1 -- 172.21.15.77:0/889827178 >> 172.21.15.77:0/889827178 conn(0x7fd99c0f74e0 msgr2=0x7fd99c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.537+0000 7fd9a3023700 1 -- 172.21.15.77:0/889827178 shutdown_connections 2024-02-19T19:01:14.556 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.537+0000 7fd9a3023700 1 -- 172.21.15.77:0/889827178 wait complete. 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.537+0000 7fd9a3023700 1 Processor -- start 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.537+0000 7fd9a3023700 1 -- start start 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd9a3023700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c0fda80 0x7fd99c10d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd9a3023700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd99c1003a0 0x7fd99c10ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd9a3023700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c10e3f0 con 0x7fd99c0fda80 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd9a3023700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c10ec90 con 0x7fd99c1003a0 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd99bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd99c1003a0 0x7fd99c10ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd9a0dbf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c0fda80 0x7fd99c10d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd99bfff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd99c1003a0 0x7fd99c10ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.77:47160/0 (socket says 172.21.15.77:47160) 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd99bfff700 1 -- 172.21.15.77:0/4240279402 learned_addr learned my addr 172.21.15.77:0/4240279402 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd99bfff700 1 -- 172.21.15.77:0/4240279402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c0fda80 msgr2=0x7fd99c10d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.557 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd99bfff700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c0fda80 0x7fd99c10d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd99bfff700 1 -- 172.21.15.77:0/4240279402 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd988005040 con 0x7fd99c1003a0 2024-02-19T19:01:14.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd9a0dbf700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c0fda80 0x7fd99c10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:01:14.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.538+0000 7fd99bfff700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd99c1003a0 0x7fd99c10ddf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd988000c00 tx=0x7fd988009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:14.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.539+0000 7fd9997fa700 1 -- 172.21.15.77:0/4240279402 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd988021070 con 0x7fd99c1003a0 2024-02-19T19:01:14.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.539+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99c10ef10 con 0x7fd99c1003a0 2024-02-19T19:01:14.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.539+0000 7fd9997fa700 1 -- 172.21.15.77:0/4240279402 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd98801c400 con 0x7fd99c1003a0 2024-02-19T19:01:14.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.539+0000 7fd9997fa700 1 -- 172.21.15.77:0/4240279402 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd98800e070 con 0x7fd99c1003a0 2024-02-19T19:01:14.558 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.539+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99c10f3f0 con 0x7fd99c1003a0 2024-02-19T19:01:14.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.540+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99c04bed0 con 0x7fd99c1003a0 2024-02-19T19:01:14.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.541+0000 7fd9997fa700 1 -- 172.21.15.77:0/4240279402 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7fd988013af0 con 0x7fd99c1003a0 2024-02-19T19:01:14.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.541+0000 7fd9997fa700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd98c06b3b0 0x7fd98c06d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.541+0000 7fd9997fa700 1 -- 172.21.15.77:0/4240279402 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fd98808af40 con 0x7fd99c1003a0 2024-02-19T19:01:14.560 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.542+0000 7fd9a0dbf700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd98c06b3b0 0x7fd98c06d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.561 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.542+0000 7fd9a0dbf700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd98c06b3b0 0x7fd98c06d860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd990004fd0 tx=0x7fd99000b040 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:14.565 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.546+0000 7fd9997fa700 1 -- 172.21.15.77:0/4240279402 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd988058ab0 con 0x7fd99c1003a0 2024-02-19T19:01:14.857 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.839+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd99c10f8c0 con 0x7fd98c06b3b0 2024-02-19T19:01:14.858 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.840+0000 7fd9997fa700 1 -- 172.21.15.77:0/4240279402 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7fd99c10f8c0 con 0x7fd98c06b3b0 2024-02-19T19:01:14.861 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.842+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd98c06b3b0 msgr2=0x7fd98c06d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.842+0000 7fd9a3023700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd98c06b3b0 0x7fd98c06d860 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd990004fd0 tx=0x7fd99000b040 comp rx=0 tx=0).stop 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.842+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd99c1003a0 msgr2=0x7fd99c10ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.842+0000 7fd9a3023700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd99c1003a0 0x7fd99c10ddf0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd988000c00 tx=0x7fd988009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.843+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 shutdown_connections 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.843+0000 7fd9a3023700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd98c06b3b0 0x7fd98c06d860 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.843+0000 7fd9a3023700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd99c0fda80 0x7fd99c10d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.843+0000 7fd9a3023700 1 --2- 172.21.15.77:0/4240279402 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd99c1003a0 0x7fd99c10ddf0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.843+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 >> 172.21.15.77:0/4240279402 conn(0x7fd99c0f74e0 msgr2=0x7fd99c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.843+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 shutdown_connections 2024-02-19T19:01:14.862 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.843+0000 7fd9a3023700 1 -- 172.21.15.77:0/4240279402 wait complete. 2024-02-19T19:01:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.956+0000 7fd82bc91700 1 -- 172.21.15.77:0/4092975317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fcfb0 msgr2=0x7fd8240fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.956+0000 7fd82bc91700 1 --2- 172.21.15.77:0/4092975317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fcfb0 0x7fd8240fd420 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd814004940 tx=0x7fd814009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.956+0000 7fd82bc91700 1 -- 172.21.15.77:0/4092975317 shutdown_connections 2024-02-19T19:01:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.956+0000 7fd82bc91700 1 --2- 172.21.15.77:0/4092975317 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fcfb0 0x7fd8240fd420 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.956+0000 7fd82bc91700 1 --2- 172.21.15.77:0/4092975317 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8240fbe40 0x7fd8240fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.956+0000 7fd82bc91700 1 -- 172.21.15.77:0/4092975317 >> 172.21.15.77:0/4092975317 conn(0x7fd8240f7500 msgr2=0x7fd8240f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.956+0000 7fd82bc91700 1 -- 172.21.15.77:0/4092975317 shutdown_connections 2024-02-19T19:01:14.975 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.956+0000 7fd82bc91700 1 -- 172.21.15.77:0/4092975317 wait complete. 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd82bc91700 1 Processor -- start 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd82bc91700 1 -- start start 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd82bc91700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fbe40 0x7fd8240fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd82bc91700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8240fcfb0 0x7fd8240fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd82bc91700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240fea10 con 0x7fd8240fbe40 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd82bc91700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8240feb80 con 0x7fd8240fcfb0 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd829a2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fbe40 0x7fd8240fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd829a2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fbe40 0x7fd8240fff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50068/0 (socket says 172.21.15.77:50068) 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd829a2d700 1 -- 172.21.15.77:0/4018558875 learned_addr learned my addr 172.21.15.77:0/4018558875 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.957+0000 7fd82922c700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8240fcfb0 0x7fd8240fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.977 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd829a2d700 1 -- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8240fcfb0 msgr2=0x7fd8240fe380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd829a2d700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8240fcfb0 0x7fd8240fe380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd829a2d700 1 -- 172.21.15.77:0/4018558875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd814005040 con 0x7fd8240fbe40 2024-02-19T19:01:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd829a2d700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fbe40 0x7fd8240fff00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd82000b790 tx=0x7fd820009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd81a7fc700 1 -- 172.21.15.77:0/4018558875 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd82000f6f0 con 0x7fd8240fbe40 2024-02-19T19:01:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8240fee60 con 0x7fd8240fbe40 2024-02-19T19:01:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd81a7fc700 1 -- 172.21.15.77:0/4018558875 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd82000fd20 con 0x7fd8240fbe40 2024-02-19T19:01:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd81a7fc700 1 -- 172.21.15.77:0/4018558875 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8200103a0 con 0x7fd8240fbe40 2024-02-19T19:01:14.978 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.958+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8240ff1e0 con 0x7fd8240fbe40 2024-02-19T19:01:14.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.959+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd82404bed0 con 0x7fd8240fbe40 2024-02-19T19:01:14.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.959+0000 7fd81a7fc700 1 -- 172.21.15.77:0/4018558875 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7fd820010500 con 0x7fd8240fbe40 2024-02-19T19:01:14.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.960+0000 7fd81a7fc700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd81006b2e0 0x7fd81006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:14.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.960+0000 7fd81a7fc700 1 -- 172.21.15.77:0/4018558875 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fd820089980 con 0x7fd8240fbe40 2024-02-19T19:01:14.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.960+0000 7fd82922c700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd81006b2e0 0x7fd81006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:14.979 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.961+0000 7fd82922c700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd81006b2e0 0x7fd81006d790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd8140043d0 tx=0x7fd814009e30 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:14.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:14.965+0000 7fd81a7fc700 1 -- 172.21.15.77:0/4018558875 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd820057810 con 0x7fd8240fbe40 2024-02-19T19:01:15.223 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.204+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fd824066660 con 0x7fd81006b2e0 2024-02-19T19:01:15.223 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:14 smithi077 ceph-mon[29247]: pgmap v23: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:15.233 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.214+0000 7fd81a7fc700 1 -- 172.21.15.77:0/4018558875 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3388 (secure 0 0 0) 0x7fd824066660 con 0x7fd81006b2e0 2024-02-19T19:01:15.233 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 starting - - - - 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (10m) 31s ago 10m 6656k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (7m) 33s ago 7m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (6m) 31s ago 8m 87.1M - 9.4.7 954c08fa6188 2cb428963663 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (2m) 31s ago 2m 13.3M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (2m) 31s ago 2m 27.1M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (2m) 33s ago 2m 13.1M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (2m) 33s ago 2m 12.2M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:9283 running (11m) 31s ago 11m 403M - 17.2.7-521-gc80e7168 10b0c032c913 78483a70884c 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283 running (53s) 33s ago 7m 488M - 19.0.0-1450-gf78a58c0 f081f847070a bc512497a554 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (11m) 31s ago 11m 51.2M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (7m) 33s ago 7m 38.2M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 starting - - - - 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (7m) 33s ago 7m 15.2M - 1.5.0 0da6a335fe13 5ec125749863 2024-02-19T19:01:15.234 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (6m) 31s ago 6m 47.7M 1557M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T19:01:15.235 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (6m) 31s ago 6m 46.7M 1557M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T19:01:15.235 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (5m) 31s ago 5m 45.7M 1557M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T19:01:15.235 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (5m) 33s ago 5m 45.9M 2581M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T19:01:15.235 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (4m) 33s ago 4m 45.9M 2581M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T19:01:15.235 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (4m) 33s ago 4m 46.0M 2581M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T19:01:15.235 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (6m) 31s ago 7m 57.3M - 2.43.0 a07b618ecd1d 6dc30fbe4b32 2024-02-19T19:01:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.217+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd81006b2e0 msgr2=0x7fd81006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.217+0000 7fd82bc91700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd81006b2e0 0x7fd81006d790 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd8140043d0 tx=0x7fd814009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.217+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fbe40 msgr2=0x7fd8240fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.217+0000 7fd82bc91700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fbe40 0x7fd8240fff00 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd82000b790 tx=0x7fd820009250 comp rx=0 tx=0).stop 2024-02-19T19:01:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.218+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 shutdown_connections 2024-02-19T19:01:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.218+0000 7fd82bc91700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fd81006b2e0 0x7fd81006d790 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.218+0000 7fd82bc91700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8240fbe40 0x7fd8240fff00 secure :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fd82000b790 tx=0x7fd820009250 comp rx=0 tx=0).stop 2024-02-19T19:01:15.237 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.218+0000 7fd82bc91700 1 --2- 172.21.15.77:0/4018558875 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8240fcfb0 0x7fd8240fe380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.218+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 >> 172.21.15.77:0/4018558875 conn(0x7fd8240f7500 msgr2=0x7fd824065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:15.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.218+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 shutdown_connections 2024-02-19T19:01:15.238 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.218+0000 7fd82bc91700 1 -- 172.21.15.77:0/4018558875 wait complete. 2024-02-19T19:01:15.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:14 smithi104 ceph-mon[42320]: pgmap v23: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:15.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.365+0000 7f4799371700 1 -- 172.21.15.77:0/3975309738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47940fda80 msgr2=0x7f47940ffe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:15.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.365+0000 7f4799371700 1 --2- 172.21.15.77:0/3975309738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47940fda80 0x7f47940ffe60 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4784004940 tx=0x7f4784009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:15.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.365+0000 7f4799371700 1 -- 172.21.15.77:0/3975309738 shutdown_connections 2024-02-19T19:01:15.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.365+0000 7f4799371700 1 --2- 172.21.15.77:0/3975309738 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47941003a0 0x7f4794102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.365+0000 7f4799371700 1 --2- 172.21.15.77:0/3975309738 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47940fda80 0x7f47940ffe60 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.384 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.365+0000 7f4799371700 1 -- 172.21.15.77:0/3975309738 >> 172.21.15.77:0/3975309738 conn(0x7f47940f74e0 msgr2=0x7f47940f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:15.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.366+0000 7f4799371700 1 -- 172.21.15.77:0/3975309738 shutdown_connections 2024-02-19T19:01:15.385 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.366+0000 7f4799371700 1 -- 172.21.15.77:0/3975309738 wait complete. 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.366+0000 7f4799371700 1 Processor -- start 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.366+0000 7f4799371700 1 -- start start 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f4799371700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47940fda80 0x7f479410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f4799371700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47941003a0 0x7f479410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f4799371700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479410e470 con 0x7f47941003a0 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f4799371700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f479410ed10 con 0x7f47940fda80 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f47927fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47941003a0 0x7f479410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f47927fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47941003a0 0x7f479410de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50080/0 (socket says 172.21.15.77:50080) 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f47927fc700 1 -- 172.21.15.77:0/2447545445 learned_addr learned my addr 172.21.15.77:0/2447545445 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f4792ffd700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47940fda80 0x7f479410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f47927fc700 1 -- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47940fda80 msgr2=0x7f479410d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:15.386 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f47927fc700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47940fda80 0x7f479410d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f47927fc700 1 -- 172.21.15.77:0/2447545445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4784005040 con 0x7f47941003a0 2024-02-19T19:01:15.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.367+0000 7f47927fc700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47941003a0 0x7f479410de70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4788002a50 tx=0x7f47880046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:15.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.368+0000 7f477b7fe700 1 -- 172.21.15.77:0/2447545445 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4788007cc0 con 0x7f47941003a0 2024-02-19T19:01:15.387 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.368+0000 7f477b7fe700 1 -- 172.21.15.77:0/2447545445 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f478800f040 con 0x7f47941003a0 2024-02-19T19:01:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.368+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f479410eff0 con 0x7f47941003a0 2024-02-19T19:01:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.368+0000 7f477b7fe700 1 -- 172.21.15.77:0/2447545445 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f478800da30 con 0x7f47941003a0 2024-02-19T19:01:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.368+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f479410f4e0 con 0x7f47941003a0 2024-02-19T19:01:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.369+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f479404bed0 con 0x7f47941003a0 2024-02-19T19:01:15.388 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.369+0000 7f477b7fe700 1 -- 172.21.15.77:0/2447545445 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7f478800db90 con 0x7f47941003a0 2024-02-19T19:01:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.370+0000 7f477b7fe700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f477c06b2e0 0x7f477c06d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.370+0000 7f477b7fe700 1 -- 172.21.15.77:0/2447545445 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f4788089bd0 con 0x7f47941003a0 2024-02-19T19:01:15.389 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.370+0000 7f4792ffd700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f477c06b2e0 0x7f477c06d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:15.390 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.371+0000 7f4792ffd700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f477c06b2e0 0x7f477c06d790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4784004f90 tx=0x7f4784009e30 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:15.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.375+0000 7f477b7fe700 1 -- 172.21.15.77:0/2447545445 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f4788057750 con 0x7f47941003a0 2024-02-19T19:01:15.778 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.759+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f4794105a10 con 0x7f47941003a0 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.760+0000 7f477b7fe700 1 -- 172.21.15.77:0/2447545445 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+846 (secure 0 0 0) 0x7f47880572e0 con 0x7f47941003a0 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 1, 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:01:15.779 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 13, 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 1 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T19:01:15.780 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f477c06b2e0 msgr2=0x7f477c06d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f477c06b2e0 0x7f477c06d790 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4784004f90 tx=0x7f4784009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47941003a0 msgr2=0x7f479410de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47941003a0 0x7f479410de70 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4788002a50 tx=0x7f47880046b0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 shutdown_connections 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f477c06b2e0 0x7f477c06d790 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f47940fda80 0x7f479410d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 --2- 172.21.15.77:0/2447545445 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f47941003a0 0x7f479410de70 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 >> 172.21.15.77:0/2447545445 conn(0x7f47940f74e0 msgr2=0x7f47940f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:15.783 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 shutdown_connections 2024-02-19T19:01:15.784 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.764+0000 7f4799371700 1 -- 172.21.15.77:0/2447545445 wait complete. 2024-02-19T19:01:15.908 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 -- 172.21.15.77:0/238868727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fcfb0 msgr2=0x7f94ec0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 --2- 172.21.15.77:0/238868727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fcfb0 0x7f94ec0fd420 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f94e0004940 tx=0x7f94e0009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 -- 172.21.15.77:0/238868727 shutdown_connections 2024-02-19T19:01:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 --2- 172.21.15.77:0/238868727 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fcfb0 0x7f94ec0fd420 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 --2- 172.21.15.77:0/238868727 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ec0fbe40 0x7f94ec0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 -- 172.21.15.77:0/238868727 >> 172.21.15.77:0/238868727 conn(0x7f94ec0f74e0 msgr2=0x7f94ec0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 -- 172.21.15.77:0/238868727 shutdown_connections 2024-02-19T19:01:15.909 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 -- 172.21.15.77:0/238868727 wait complete. 2024-02-19T19:01:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.890+0000 7f94f14d4700 1 Processor -- start 2024-02-19T19:01:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94f14d4700 1 -- start start 2024-02-19T19:01:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94f14d4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fbe40 0x7f94ec110670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94f14d4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ec0fcfb0 0x7f94ec112bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94f14d4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec0664f0 con 0x7f94ec0fbe40 2024-02-19T19:01:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94f14d4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec066630 con 0x7f94ec0fcfb0 2024-02-19T19:01:15.910 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94eaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fbe40 0x7f94ec110670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94eaffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fbe40 0x7f94ec110670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:50094/0 (socket says 172.21.15.77:50094) 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94eaffd700 1 -- 172.21.15.77:0/3493321397 learned_addr learned my addr 172.21.15.77:0/3493321397 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94ea7fc700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ec0fcfb0 0x7f94ec112bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94eaffd700 1 -- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ec0fcfb0 msgr2=0x7f94ec112bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94eaffd700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ec0fcfb0 0x7f94ec112bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94eaffd700 1 -- 172.21.15.77:0/3493321397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0005040 con 0x7f94ec0fbe40 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94ea7fc700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ec0fcfb0 0x7f94ec112bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.891+0000 7f94eaffd700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fbe40 0x7f94ec110670 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00b790 tx=0x7f94dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.892+0000 7f94d37fe700 1 -- 172.21.15.77:0/3493321397 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94dc00f6f0 con 0x7f94ec0fbe40 2024-02-19T19:01:15.911 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.892+0000 7f94d37fe700 1 -- 172.21.15.77:0/3493321397 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f94dc00fd20 con 0x7f94ec0fbe40 2024-02-19T19:01:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.892+0000 7f94d37fe700 1 -- 172.21.15.77:0/3493321397 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f94dc0103a0 con 0x7f94ec0fbe40 2024-02-19T19:01:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.892+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ec113250 con 0x7f94ec0fbe40 2024-02-19T19:01:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.892+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ec113730 con 0x7f94ec0fbe40 2024-02-19T19:01:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.893+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ec04bed0 con 0x7f94ec0fbe40 2024-02-19T19:01:15.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.894+0000 7f94d37fe700 1 -- 172.21.15.77:0/3493321397 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7f94dc00f850 con 0x7f94ec0fbe40 2024-02-19T19:01:15.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.894+0000 7f94d37fe700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f94d406b2e0 0x7f94d406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:15.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.894+0000 7f94d37fe700 1 -- 172.21.15.77:0/3493321397 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f94dc089a20 con 0x7f94ec0fbe40 2024-02-19T19:01:15.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.895+0000 7f94ea7fc700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f94d406b2e0 0x7f94d406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:15.914 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.895+0000 7f94ea7fc700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f94d406b2e0 0x7f94d406d790 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f94e000b3b0 tx=0x7f94e0009e30 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:15.918 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:15.899+0000 7f94d37fe700 1 -- 172.21.15.77:0/3493321397 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f94dc053c30 con 0x7f94ec0fbe40 2024-02-19T19:01:16.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:15 smithi077 ceph-mon[29247]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:16.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:15 smithi077 ceph-mon[29247]: from='client.24411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:16.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:15 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/2447545445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:01:16.146 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.128+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f94ec113a60 con 0x7f94d406b2e0 2024-02-19T19:01:16.148 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.129+0000 7f94d37fe700 1 -- 172.21.15.77:0/3493321397 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+310 (secure 0 0 0) 0x7f94ec113a60 con 0x7f94d406b2e0 2024-02-19T19:01:16.148 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:01:16.148 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:01:16.148 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T19:01:16.148 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T19:01:16.148 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [], 2024-02-19T19:01:16.148 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "1/2 daemons upgraded", 2024-02-19T19:01:16.149 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "", 2024-02-19T19:01:16.149 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T19:01:16.149 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:01:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f94d406b2e0 msgr2=0x7f94d406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f94d406b2e0 0x7f94d406d790 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f94e000b3b0 tx=0x7f94e0009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fbe40 msgr2=0x7f94ec110670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fbe40 0x7f94ec110670 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00b790 tx=0x7f94dc009250 comp rx=0 tx=0).stop 2024-02-19T19:01:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 shutdown_connections 2024-02-19T19:01:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f94d406b2e0 0x7f94d406d790 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:16.150 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f94ec0fbe40 0x7f94ec110670 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 --2- 172.21.15.77:0/3493321397 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f94ec0fcfb0 0x7f94ec112bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 >> 172.21.15.77:0/3493321397 conn(0x7f94ec0f74e0 msgr2=0x7f94ec100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 shutdown_connections 2024-02-19T19:01:16.151 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:16.131+0000 7f94f14d4700 1 -- 172.21.15.77:0/3493321397 wait complete. 2024-02-19T19:01:16.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:15 smithi104 ceph-mon[42320]: from='client.14694 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:16.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:15 smithi104 ceph-mon[42320]: from='client.24411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:16.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:15 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/2447545445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:01:17.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:16 smithi077 ceph-mon[29247]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:17.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:16 smithi077 ceph-mon[29247]: pgmap v24: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:17.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:16 smithi077 ceph-mon[29247]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:17.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:16 smithi104 ceph-mon[42320]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:17.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:16 smithi104 ceph-mon[42320]: pgmap v24: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:17.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:16 smithi104 ceph-mon[42320]: from='client.14710 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:19.226 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:18 smithi077 ceph-mon[29247]: pgmap v25: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:19.227 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:18 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:19.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:18 smithi104 ceph-mon[42320]: pgmap v25: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:19.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:18 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:21.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:20 smithi077 ceph-mon[29247]: pgmap v26: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:21.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:20 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:21.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:20 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:20 smithi104 ceph-mon[42320]: pgmap v26: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:20 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:21.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:20 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:22.203 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:21 smithi077 ceph-mon[29247]: Reconfiguring prometheus.smithi077 (dependencies changed)... 2024-02-19T19:01:22.203 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:21 smithi077 ceph-mon[29247]: Reconfiguring daemon prometheus.smithi077 on smithi077 2024-02-19T19:01:22.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:21 smithi104 ceph-mon[42320]: Reconfiguring prometheus.smithi077 (dependencies changed)... 2024-02-19T19:01:22.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:21 smithi104 ceph-mon[42320]: Reconfiguring daemon prometheus.smithi077 on smithi077 2024-02-19T19:01:23.175 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:22 smithi077 ceph-mon[29247]: pgmap v27: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:23.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:22 smithi104 ceph-mon[42320]: pgmap v27: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:25.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:24 smithi077 ceph-mon[29247]: pgmap v28: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:25.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:24 smithi104 ceph-mon[42320]: pgmap v28: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:26 smithi104 ceph-mon[42320]: pgmap v29: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:26 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:26 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:27.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:26 smithi077 ceph-mon[29247]: pgmap v29: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:27.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:26 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:27.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:26 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:28.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:27 smithi077 ceph-mon[29247]: Reconfiguring node-exporter.smithi104 (dependencies changed)... 2024-02-19T19:01:28.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:27 smithi077 ceph-mon[29247]: Reconfiguring daemon node-exporter.smithi104 on smithi104 2024-02-19T19:01:28.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:27 smithi104 ceph-mon[42320]: Reconfiguring node-exporter.smithi104 (dependencies changed)... 2024-02-19T19:01:28.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:27 smithi104 ceph-mon[42320]: Reconfiguring daemon node-exporter.smithi104 on smithi104 2024-02-19T19:01:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:28 smithi077 ceph-mon[29247]: pgmap v30: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:28 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:28 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:28 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T19:01:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:28 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T19:01:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:28 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:01:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:28 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:29.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:28 smithi104 ceph-mon[42320]: pgmap v30: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:29.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:28 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:29.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:28 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:29.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:28 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T19:01:29.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:28 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T19:01:29.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:28 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:01:29.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:28 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:30.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:29 smithi077 ceph-mon[29247]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T19:01:30.182 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:29 smithi077 ceph-mon[29247]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T19:01:30.182 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:29 smithi077 ceph-mon[29247]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:01:30.182 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:29 smithi077 ceph-mon[29247]: Upgrade: Need to upgrade myself (mgr.smithi104.sdpzmi) 2024-02-19T19:01:30.182 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:29 smithi077 ceph-mon[29247]: Upgrade: Need to upgrade myself (mgr.smithi104.sdpzmi) 2024-02-19T19:01:30.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:29 smithi104 ceph-mon[42320]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T19:01:30.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:29 smithi104 ceph-mon[42320]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-02-19T19:01:30.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:29 smithi104 ceph-mon[42320]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:01:30.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:29 smithi104 ceph-mon[42320]: Upgrade: Need to upgrade myself (mgr.smithi104.sdpzmi) 2024-02-19T19:01:30.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:29 smithi104 ceph-mon[42320]: Upgrade: Need to upgrade myself (mgr.smithi104.sdpzmi) 2024-02-19T19:01:31.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:30 smithi077 ceph-mon[29247]: pgmap v31: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:31.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:30 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:31.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:30 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.yiiqpd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:01:31.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:30 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.yiiqpd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:01:31.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:30 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:01:31.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:30 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:01:31.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:30 smithi104 ceph-mon[42320]: pgmap v31: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:31.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:30 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:31.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:30 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.yiiqpd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:01:31.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:30 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi077.yiiqpd", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:01:31.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:30 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:01:31.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:30 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:01:32.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:31 smithi077 ceph-mon[29247]: Upgrade: Updating mgr.smithi077.yiiqpd 2024-02-19T19:01:32.059 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:31 smithi077 ceph-mon[29247]: Deploying daemon mgr.smithi077.yiiqpd on smithi077 2024-02-19T19:01:32.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:31 smithi104 ceph-mon[42320]: Upgrade: Updating mgr.smithi077.yiiqpd 2024-02-19T19:01:32.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:31 smithi104 ceph-mon[42320]: Deploying daemon mgr.smithi077.yiiqpd on smithi077 2024-02-19T19:01:33.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:32 smithi077 ceph-mon[29247]: pgmap v32: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:33.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:32 smithi104 ceph-mon[42320]: pgmap v32: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:34.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:33 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:34.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:33 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:35.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:34 smithi077 ceph-mon[29247]: pgmap v33: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:35.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:34 smithi104 ceph-mon[42320]: pgmap v33: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:37.167 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:36 smithi077 ceph-mon[29247]: pgmap v34: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:37.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:36 smithi104 ceph-mon[42320]: pgmap v34: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:39.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:38 smithi077 ceph-mon[29247]: pgmap v35: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:39.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:38 smithi104 ceph-mon[42320]: pgmap v35: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:40.306 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:40 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:40.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:40 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:40.307 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:40 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:40.428 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:40 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:40.428 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:40 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:40.428 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:40 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:41.251 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:41 smithi077 ceph-mon[29247]: pgmap v36: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:41.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:41 smithi104 ceph-mon[42320]: pgmap v36: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:43.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:43 smithi077 ceph-mon[29247]: pgmap v37: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:43.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:43 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:43.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:43 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:43.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:43 smithi104 ceph-mon[42320]: pgmap v37: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:43.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:43 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:43.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:43 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:44.480 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:44 smithi077 ceph-mon[29247]: pgmap v38: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:44.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:44 smithi104 ceph-mon[42320]: pgmap v38: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:46.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.242+0000 7f2211858700 1 -- 172.21.15.77:0/3024344872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c0fbce0 msgr2=0x7f220c102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.242+0000 7f2211858700 1 --2- 172.21.15.77:0/3024344872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c0fbce0 0x7f220c102560 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2200004940 tx=0x7f2200009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:46.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:45 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:46.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:45 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.242+0000 7f2211858700 1 -- 172.21.15.77:0/3024344872 shutdown_connections 2024-02-19T19:01:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.242+0000 7f2211858700 1 --2- 172.21.15.77:0/3024344872 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c0fbce0 0x7f220c102560 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.242+0000 7f2211858700 1 --2- 172.21.15.77:0/3024344872 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220c0fb300 0x7f220c0fb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.242+0000 7f2211858700 1 -- 172.21.15.77:0/3024344872 >> 172.21.15.77:0/3024344872 conn(0x7f220c0f6ca0 msgr2=0x7f220c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.242+0000 7f2211858700 1 -- 172.21.15.77:0/3024344872 shutdown_connections 2024-02-19T19:01:46.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.242+0000 7f2211858700 1 -- 172.21.15.77:0/3024344872 wait complete. 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f2211858700 1 Processor -- start 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f2211858700 1 -- start start 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f2211858700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220c0fb300 0x7f220c10da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f2211858700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c10df50 0x7f220c10edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f2211858700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c10e410 con 0x7f220c10df50 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f2211858700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220c10e550 con 0x7f220c0fb300 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f220a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c10df50 0x7f220c10edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f220a7fc700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c10df50 0x7f220c10edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:58384/0 (socket says 172.21.15.77:58384) 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.243+0000 7f220a7fc700 1 -- 172.21.15.77:0/1986950411 learned_addr learned my addr 172.21.15.77:0/1986950411 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f220affd700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220c0fb300 0x7f220c10da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f220a7fc700 1 -- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220c0fb300 msgr2=0x7f220c10da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f220a7fc700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220c0fb300 0x7f220c10da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f220a7fc700 1 -- 172.21.15.77:0/1986950411 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2200005040 con 0x7f220c10df50 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f220affd700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220c0fb300 0x7f220c10da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-19T19:01:46.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f220a7fc700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c10df50 0x7f220c10edf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2200009150 tx=0x7f2200009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f21f37fe700 1 -- 172.21.15.77:0/1986950411 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220000e040 con 0x7f220c10df50 2024-02-19T19:01:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f21f37fe700 1 -- 172.21.15.77:0/1986950411 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f220000bbf0 con 0x7f220c10df50 2024-02-19T19:01:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f21f37fe700 1 -- 172.21.15.77:0/1986950411 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f220001c630 con 0x7f220c10df50 2024-02-19T19:01:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220c10f3f0 con 0x7f220c10df50 2024-02-19T19:01:46.265 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.244+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220c10f900 con 0x7f220c10df50 2024-02-19T19:01:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.245+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220c107a90 con 0x7f220c10df50 2024-02-19T19:01:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.246+0000 7f21f37fe700 1 -- 172.21.15.77:0/1986950411 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7f220000bd60 con 0x7f220c10df50 2024-02-19T19:01:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.247+0000 7f21f37fe700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f21f406b2e0 0x7f21f406d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.247+0000 7f21f37fe700 1 -- 172.21.15.77:0/1986950411 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f220008b010 con 0x7f220c10df50 2024-02-19T19:01:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.247+0000 7f220affd700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f21f406b2e0 0x7f21f406d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.266 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.247+0000 7f220affd700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f21f406b2e0 0x7f21f406d790 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009fd0 tx=0x7f21fc009f00 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:46.274 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.255+0000 7f21f37fe700 1 -- 172.21.15.77:0/1986950411 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f2200058c40 con 0x7f220c10df50 2024-02-19T19:01:46.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:45 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:46.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:45 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:46.523 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.504+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f220c0fbce0 con 0x7f21f406b2e0 2024-02-19T19:01:46.524 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.505+0000 7f21f37fe700 1 -- 172.21.15.77:0/1986950411 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f220c0fbce0 con 0x7f21f406b2e0 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f21f406b2e0 msgr2=0x7f21f406d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f21f406b2e0 0x7f21f406d790 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009fd0 tx=0x7f21fc009f00 comp rx=0 tx=0).stop 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c10df50 msgr2=0x7f220c10edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c10df50 0x7f220c10edf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2200009150 tx=0x7f2200009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 shutdown_connections 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f21f406b2e0 0x7f21f406d790 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f220c0fb300 0x7f220c10da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 --2- 172.21.15.77:0/1986950411 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f220c10df50 0x7f220c10edf0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.507+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 >> 172.21.15.77:0/1986950411 conn(0x7f220c0f6ca0 msgr2=0x7f220c100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.508+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 shutdown_connections 2024-02-19T19:01:46.527 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.508+0000 7f2211858700 1 -- 172.21.15.77:0/1986950411 wait complete. 2024-02-19T19:01:46.539 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T19:01:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 -- 172.21.15.77:0/1217243395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 msgr2=0x7f84380fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 --2- 172.21.15.77:0/1217243395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 0x7f84380fd320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f84280048b0 tx=0x7f8428009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 -- 172.21.15.77:0/1217243395 shutdown_connections 2024-02-19T19:01:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 --2- 172.21.15.77:0/1217243395 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84380fe0e0 0x7f84380fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 --2- 172.21.15.77:0/1217243395 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 0x7f84380fd320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 -- 172.21.15.77:0/1217243395 >> 172.21.15.77:0/1217243395 conn(0x7f84380f8620 msgr2=0x7f84380faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 -- 172.21.15.77:0/1217243395 shutdown_connections 2024-02-19T19:01:46.647 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 -- 172.21.15.77:0/1217243395 wait complete. 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 Processor -- start 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.628+0000 7f843ca17700 1 -- start start 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f843ca17700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 0x7f843810d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f843ca17700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84380fe0e0 0x7f843810de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f843ca17700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843810e420 con 0x7f84380fe0e0 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f843ca17700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f843810ecc0 con 0x7f84380fcef0 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8435d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84380fe0e0 0x7f843810de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f843659c700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 0x7f843810d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8435d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84380fe0e0 0x7f843810de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52544/0 (socket says 172.21.15.77:52544) 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8435d9b700 1 -- 172.21.15.77:0/2906873716 learned_addr learned my addr 172.21.15.77:0/2906873716 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8435d9b700 1 -- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 msgr2=0x7f843810d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8435d9b700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 0x7f843810d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8435d9b700 1 -- 172.21.15.77:0/2906873716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f842c005040 con 0x7f84380fe0e0 2024-02-19T19:01:46.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f843659c700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 0x7f843810d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:01:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8435d9b700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84380fe0e0 0x7f843810de20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f842c0096b0 tx=0x7f842c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8426ffd700 1 -- 172.21.15.77:0/2906873716 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f842c017070 con 0x7f84380fe0e0 2024-02-19T19:01:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8426ffd700 1 -- 172.21.15.77:0/2906873716 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f842c00f040 con 0x7f84380fe0e0 2024-02-19T19:01:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8428005040 con 0x7f84380fe0e0 2024-02-19T19:01:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f8426ffd700 1 -- 172.21.15.77:0/2906873716 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f842c013be0 con 0x7f84380fe0e0 2024-02-19T19:01:46.650 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.629+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f843810f2f0 con 0x7f84380fe0e0 2024-02-19T19:01:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.630+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f843804bed0 con 0x7f84380fe0e0 2024-02-19T19:01:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.631+0000 7f8426ffd700 1 -- 172.21.15.77:0/2906873716 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7f842c00e840 con 0x7f84380fe0e0 2024-02-19T19:01:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.631+0000 7f8426ffd700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f842006b2e0 0x7f842006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.631+0000 7f8426ffd700 1 -- 172.21.15.77:0/2906873716 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f842c089c10 con 0x7f84380fe0e0 2024-02-19T19:01:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.632+0000 7f843659c700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f842006b2e0 0x7f842006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.651 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.632+0000 7f843659c700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f842006b2e0 0x7f842006d790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8428013740 tx=0x7f8428009e30 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:46.655 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.636+0000 7f8426ffd700 1 -- 172.21.15.77:0/2906873716 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f842c057720 con 0x7f84380fe0e0 2024-02-19T19:01:46.878 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.859+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f843810f740 con 0x7f842006b2e0 2024-02-19T19:01:46.879 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.860+0000 7f8426ffd700 1 -- 172.21.15.77:0/2906873716 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f843810f740 con 0x7f842006b2e0 2024-02-19T19:01:46.881 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f842006b2e0 msgr2=0x7f842006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f842006b2e0 0x7f842006d790 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8428013740 tx=0x7f8428009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84380fe0e0 msgr2=0x7f843810de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84380fe0e0 0x7f843810de20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f842c0096b0 tx=0x7f842c009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 shutdown_connections 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f842006b2e0 0x7f842006d790 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f84380fcef0 0x7f843810d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 --2- 172.21.15.77:0/2906873716 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f84380fe0e0 0x7f843810de20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 >> 172.21.15.77:0/2906873716 conn(0x7f84380f8620 msgr2=0x7f84381013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 shutdown_connections 2024-02-19T19:01:46.882 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.862+0000 7f843ca17700 1 -- 172.21.15.77:0/2906873716 wait complete. 2024-02-19T19:01:46.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 -- 172.21.15.77:0/2457577788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fcef0 msgr2=0x7fe8840fd320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 --2- 172.21.15.77:0/2457577788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fcef0 0x7fe8840fd320 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009ce0 tx=0x7fe87c004d10 comp rx=0 tx=0).stop 2024-02-19T19:01:46.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 -- 172.21.15.77:0/2457577788 shutdown_connections 2024-02-19T19:01:46.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 --2- 172.21.15.77:0/2457577788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8840fe0e0 0x7fe8840fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 --2- 172.21.15.77:0/2457577788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fcef0 0x7fe8840fd320 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.986 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 -- 172.21.15.77:0/2457577788 >> 172.21.15.77:0/2457577788 conn(0x7fe8840f8620 msgr2=0x7fe8840faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:46.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 -- 172.21.15.77:0/2457577788 shutdown_connections 2024-02-19T19:01:46.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 -- 172.21.15.77:0/2457577788 wait complete. 2024-02-19T19:01:46.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 Processor -- start 2024-02-19T19:01:46.987 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.967+0000 7fe889db0700 1 -- start start 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe889db0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8840fcef0 0x7fe884071e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe889db0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fe0e0 0x7fe8840702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe889db0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840708e0 con 0x7fe8840fe0e0 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe889db0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe884070a50 con 0x7fe8840fcef0 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe882ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fe0e0 0x7fe8840702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe882ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fe0e0 0x7fe8840702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52570/0 (socket says 172.21.15.77:52570) 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe882ffd700 1 -- 172.21.15.77:0/2662810938 learned_addr learned my addr 172.21.15.77:0/2662810938 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe8837fe700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8840fcef0 0x7fe884071e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe882ffd700 1 -- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8840fcef0 msgr2=0x7fe884071e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe882ffd700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8840fcef0 0x7fe884071e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe882ffd700 1 -- 172.21.15.77:0/2662810938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009940 con 0x7fe8840fe0e0 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe882ffd700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fe0e0 0x7fe8840702e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00eb40 tx=0x7fe86c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe87bfff700 1 -- 172.21.15.77:0/2662810938 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe86c007cc0 con 0x7fe8840fe0e0 2024-02-19T19:01:46.989 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe884070cd0 con 0x7fe8840fe0e0 2024-02-19T19:01:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe87bfff700 1 -- 172.21.15.77:0/2662810938 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe86c00f040 con 0x7fe8840fe0e0 2024-02-19T19:01:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe87bfff700 1 -- 172.21.15.77:0/2662810938 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe86c00ca10 con 0x7fe8840fe0e0 2024-02-19T19:01:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.968+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8841159a0 con 0x7fe8840fe0e0 2024-02-19T19:01:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.970+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe88404bed0 con 0x7fe8840fe0e0 2024-02-19T19:01:46.990 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.970+0000 7fe87bfff700 1 -- 172.21.15.77:0/2662810938 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 24) v1 ==== 89018+0+0 (secure 0 0 0) 0x7fe86c00cc80 con 0x7fe8840fe0e0 2024-02-19T19:01:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.970+0000 7fe87bfff700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fe87006b2e0 0x7fe87006d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.971+0000 7fe87bfff700 1 -- 172.21.15.77:0/2662810938 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7fe86c0899b0 con 0x7fe8840fe0e0 2024-02-19T19:01:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.971+0000 7fe8837fe700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fe87006b2e0 0x7fe87006d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:46.991 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.971+0000 7fe8837fe700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fe87006b2e0 0x7fe87006d790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009cb0 tx=0x7fe87c004d10 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:46.994 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:46.975+0000 7fe87bfff700 1 -- 172.21.15.77:0/2662810938 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fe86c04f620 con 0x7fe8840fe0e0 2024-02-19T19:01:47.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.066+0000 7fe87bfff700 1 -- 172.21.15.77:0/2662810938 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mgrmap(e 25) v1 ==== 91311+0+0 (secure 0 0 0) 0x7fe86c087eb0 con 0x7fe8840fe0e0 2024-02-19T19:01:47.212 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.193+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fe884102ac0 con 0x7fe87006b2e0 2024-02-19T19:01:47.212 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: pgmap v39: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:47.212 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:47.212 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:47.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: Standby manager daemon smithi077.yiiqpd restarted 2024-02-19T19:01:47.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: Standby manager daemon smithi077.yiiqpd started 2024-02-19T19:01:47.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.yiiqpd/crt"}]: dispatch 2024-02-19T19:01:47.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:01:47.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.yiiqpd/key"}]: dispatch 2024-02-19T19:01:47.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:47 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:01:47.222 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.204+0000 7fe87bfff700 1 -- 172.21.15.77:0/2662810938 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7fe884102ac0 con 0x7fe87006b2e0 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (50s) 2s ago 10m 16.7M - 0.25.0 c8568f914cd2 46ae75dc6421 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (10m) 2s ago 10m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (8m) 4s ago 8m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (27s) 2s ago 8m 77.9M - 9.4.7 954c08fa6188 454ae3504841 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (3m) 2s ago 3m 13.6M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (3m) 2s ago 3m 27.4M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (3m) 4s ago 3m 13.5M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (3m) 4s ago 3m 12.5M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:8443,9283,8765 running (8s) 2s ago 11m 66.4M - 19.0.0-1450-gf78a58c0 f081f847070a e536fa0dfb57 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283 running (85s) 4s ago 7m 498M - 19.0.0-1450-gf78a58c0 f081f847070a bc512497a554 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (11m) 2s ago 12m 53.3M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T19:01:47.223 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (7m) 4s ago 7m 39.6M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (55s) 2s ago 10m 7972k - 1.5.0 0da6a335fe13 e8ff8e3e80d6 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (18s) 4s ago 7m 7926k - 1.5.0 0da6a335fe13 8230c18c1dd0 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (7m) 2s ago 7m 48.0M 1557M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (6m) 2s ago 6m 47.0M 1557M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (6m) 2s ago 6m 45.8M 1557M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (5m) 4s ago 5m 46.4M 2581M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (5m) 4s ago 5m 46.1M 2581M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (4m) 4s ago 4m 46.4M 2581M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T19:01:47.224 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (21s) 2s ago 8m 38.9M - 2.43.0 a07b618ecd1d f46d6a233db9 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fe87006b2e0 msgr2=0x7fe87006d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fe87006b2e0 0x7fe87006d790 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fe87c009cb0 tx=0x7fe87c004d10 comp rx=0 tx=0).stop 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fe0e0 msgr2=0x7fe8840702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fe0e0 0x7fe8840702e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00eb40 tx=0x7fe86c00b040 comp rx=0 tx=0).stop 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 shutdown_connections 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7fe87006b2e0 0x7fe87006d790 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fe8840fcef0 0x7fe884071e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 --2- 172.21.15.77:0/2662810938 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fe8840fe0e0 0x7fe8840702e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 >> 172.21.15.77:0/2662810938 conn(0x7fe8840f8620 msgr2=0x7fe8841013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 shutdown_connections 2024-02-19T19:01:47.226 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.206+0000 7fe889db0700 1 -- 172.21.15.77:0/2662810938 wait complete. 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.308+0000 7f58768cd700 1 -- 172.21.15.77:0/3318191035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fcfb0 msgr2=0x7f58700fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.308+0000 7f58768cd700 1 --2- 172.21.15.77:0/3318191035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fcfb0 0x7f58700fd420 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5864004940 tx=0x7f5864009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.308+0000 7f58768cd700 1 -- 172.21.15.77:0/3318191035 shutdown_connections 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.308+0000 7f58768cd700 1 --2- 172.21.15.77:0/3318191035 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fcfb0 0x7f58700fd420 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.308+0000 7f58768cd700 1 --2- 172.21.15.77:0/3318191035 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58700fbe40 0x7f58700fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.308+0000 7f58768cd700 1 -- 172.21.15.77:0/3318191035 >> 172.21.15.77:0/3318191035 conn(0x7f58700f7520 msgr2=0x7f58700f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.308+0000 7f58768cd700 1 -- 172.21.15.77:0/3318191035 shutdown_connections 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.308+0000 7f58768cd700 1 -- 172.21.15.77:0/3318191035 wait complete. 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f58768cd700 1 Processor -- start 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f58768cd700 1 -- start start 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f58768cd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fbe40 0x7f587010d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f58768cd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58700fcfb0 0x7f587010dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:47.329 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f58768cd700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587010e3e0 con 0x7f58700fbe40 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f58768cd700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587010ec80 con 0x7f58700fcfb0 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f586ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fbe40 0x7f587010d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f586ffff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fbe40 0x7f587010d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52582/0 (socket says 172.21.15.77:52582) 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f586ffff700 1 -- 172.21.15.77:0/3804838392 learned_addr learned my addr 172.21.15.77:0/3804838392 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f586f7fe700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58700fcfb0 0x7f587010dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f586ffff700 1 -- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58700fcfb0 msgr2=0x7f587010dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f586ffff700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58700fcfb0 0x7f587010dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.309+0000 7f586ffff700 1 -- 172.21.15.77:0/3804838392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5864005040 con 0x7f58700fbe40 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.310+0000 7f586ffff700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fbe40 0x7f587010d8a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5860002a80 tx=0x7f58600046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.310+0000 7f586cff9700 1 -- 172.21.15.77:0/3804838392 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5860007cc0 con 0x7f58700fbe40 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.310+0000 7f586cff9700 1 -- 172.21.15.77:0/3804838392 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f586000f040 con 0x7f58700fbe40 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.310+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587010ef60 con 0x7f58700fbe40 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.310+0000 7f586cff9700 1 -- 172.21.15.77:0/3804838392 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f586000dac0 con 0x7f58700fbe40 2024-02-19T19:01:47.330 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.310+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587010f4a0 con 0x7f58700fbe40 2024-02-19T19:01:47.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.311+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587004bed0 con 0x7f58700fbe40 2024-02-19T19:01:47.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.311+0000 7f586cff9700 1 -- 172.21.15.77:0/3804838392 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 91311+0+0 (secure 0 0 0) 0x7f586000dc20 con 0x7f58700fbe40 2024-02-19T19:01:47.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.312+0000 7f586cff9700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f585806d200 0x7f585806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:47.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.312+0000 7f586cff9700 1 -- 172.21.15.77:0/3804838392 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f586008c340 con 0x7f58700fbe40 2024-02-19T19:01:47.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.312+0000 7f586f7fe700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f585806d200 0x7f585806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:47.331 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.312+0000 7f586f7fe700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f585806d200 0x7f585806f6b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5864004f90 tx=0x7f5864009e30 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:47.335 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.316+0000 7f586cff9700 1 -- 172.21.15.77:0/3804838392 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f5860059ed0 con 0x7f58700fbe40 2024-02-19T19:01:47.477 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: pgmap v39: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:47.477 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:47.477 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:47.477 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: Standby manager daemon smithi077.yiiqpd restarted 2024-02-19T19:01:47.477 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: Standby manager daemon smithi077.yiiqpd started 2024-02-19T19:01:47.478 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.yiiqpd/crt"}]: dispatch 2024-02-19T19:01:47.478 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:01:47.478 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi077.yiiqpd/key"}]: dispatch 2024-02-19T19:01:47.478 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:47 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:01:47.624 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.605+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f587010f910 con 0x7f58700fbe40 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.606+0000 7f586cff9700 1 -- 172.21.15.77:0/3804838392 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f5860060020 con 0x7f58700fbe40 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:01:47.625 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 12, 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T19:01:47.626 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f585806d200 msgr2=0x7f585806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f585806d200 0x7f585806f6b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5864004f90 tx=0x7f5864009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fbe40 msgr2=0x7f587010d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fbe40 0x7f587010d8a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5860002a80 tx=0x7f58600046b0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 shutdown_connections 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f585806d200 0x7f585806f6b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f58700fbe40 0x7f587010d8a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 --2- 172.21.15.77:0/3804838392 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f58700fcfb0 0x7f587010dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 >> 172.21.15.77:0/3804838392 conn(0x7f58700f7520 msgr2=0x7f5870065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 shutdown_connections 2024-02-19T19:01:47.627 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.608+0000 7f58768cd700 1 -- 172.21.15.77:0/3804838392 wait complete. 2024-02-19T19:01:47.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.710+0000 7f324364e700 1 -- 172.21.15.77:0/2096961252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c070860 msgr2=0x7f323c070cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.710+0000 7f324364e700 1 --2- 172.21.15.77:0/2096961252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c070860 0x7f323c070cd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f322c004940 tx=0x7f322c009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:47.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.710+0000 7f324364e700 1 -- 172.21.15.77:0/2096961252 shutdown_connections 2024-02-19T19:01:47.729 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.710+0000 7f324364e700 1 --2- 172.21.15.77:0/2096961252 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c070860 0x7f323c070cd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.710+0000 7f324364e700 1 --2- 172.21.15.77:0/2096961252 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f323c071eb0 0x7f323c070320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.710+0000 7f324364e700 1 -- 172.21.15.77:0/2096961252 >> 172.21.15.77:0/2096961252 conn(0x7f323c0f8620 msgr2=0x7f323c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:47.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.710+0000 7f324364e700 1 -- 172.21.15.77:0/2096961252 shutdown_connections 2024-02-19T19:01:47.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.710+0000 7f324364e700 1 -- 172.21.15.77:0/2096961252 wait complete. 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f324364e700 1 Processor -- start 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f324364e700 1 -- start start 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f324364e700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f323c070860 0x7f323c10fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f324364e700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c071eb0 0x7f323c110020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f324364e700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c110620 con 0x7f323c071eb0 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f324364e700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f323c110ec0 con 0x7f323c070860 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f3240be9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c071eb0 0x7f323c110020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f3240be9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c071eb0 0x7f323c110020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:52594/0 (socket says 172.21.15.77:52594) 2024-02-19T19:01:47.731 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f3240be9700 1 -- 172.21.15.77:0/3488271845 learned_addr learned my addr 172.21.15.77:0/3488271845 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f32413ea700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f323c070860 0x7f323c10fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f3240be9700 1 -- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f323c070860 msgr2=0x7f323c10fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f3240be9700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f323c070860 0x7f323c10fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.711+0000 7f3240be9700 1 -- 172.21.15.77:0/3488271845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c005040 con 0x7f323c071eb0 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.712+0000 7f3240be9700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c071eb0 0x7f323c110020 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f322c000c00 tx=0x7f322c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.712+0000 7f3231ffb700 1 -- 172.21.15.77:0/3488271845 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c021070 con 0x7f323c071eb0 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.712+0000 7f3231ffb700 1 -- 172.21.15.77:0/3488271845 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f322c01c400 con 0x7f323c071eb0 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.712+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f323c111140 con 0x7f323c071eb0 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.712+0000 7f3231ffb700 1 -- 172.21.15.77:0/3488271845 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f322c00e070 con 0x7f323c071eb0 2024-02-19T19:01:47.732 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.712+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f323c111620 con 0x7f323c071eb0 2024-02-19T19:01:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.713+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f323c04bed0 con 0x7f323c071eb0 2024-02-19T19:01:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.713+0000 7f3231ffb700 1 -- 172.21.15.77:0/3488271845 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 25) v1 ==== 91311+0+0 (secure 0 0 0) 0x7f322c01ca50 con 0x7f323c071eb0 2024-02-19T19:01:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.714+0000 7f3231ffb700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f322806d200 0x7f322806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:01:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.714+0000 7f3231ffb700 1 -- 172.21.15.77:0/3488271845 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5722+0+0 (secure 0 0 0) 0x7f322c08d800 con 0x7f323c071eb0 2024-02-19T19:01:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.714+0000 7f32413ea700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f322806d200 0x7f322806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:01:47.733 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.714+0000 7f32413ea700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f322806d200 0x7f322806f6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3238009b80 tx=0x7f323800b040 comp rx=0 tx=0).ready entity=mgr.24355 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:01:47.737 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.718+0000 7f3231ffb700 1 -- 172.21.15.77:0/3488271845 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f322c05b390 con 0x7f323c071eb0 2024-02-19T19:01:47.965 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.946+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 --> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f323c111a30 con 0x7f322806d200 2024-02-19T19:01:47.966 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.947+0000 7f3231ffb700 1 -- 172.21.15.77:0/3488271845 <== mgr.24355 v2:172.21.15.104:6828/719997867 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f323c111a30 con 0x7f322806d200 2024-02-19T19:01:47.966 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:01:47.966 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:01:47.966 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T19:01:47.966 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T19:01:47.966 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-02-19T19:01:47.967 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-02-19T19:01:47.967 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-02-19T19:01:47.967 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-02-19T19:01:47.967 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading mgr daemons", 2024-02-19T19:01:47.967 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T19:01:47.967 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f322806d200 msgr2=0x7f322806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f322806d200 0x7f322806f6b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3238009b80 tx=0x7f323800b040 comp rx=0 tx=0).stop 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c071eb0 msgr2=0x7f323c110020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c071eb0 0x7f323c110020 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f322c000c00 tx=0x7f322c009e30 comp rx=0 tx=0).stop 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 shutdown_connections 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:6828/719997867,v1:172.21.15.104:6829/719997867] conn(0x7f322806d200 0x7f322806f6b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f323c070860 0x7f323c10fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 --2- 172.21.15.77:0/3488271845 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f323c071eb0 0x7f323c110020 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:01:47.968 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 >> 172.21.15.77:0/3488271845 conn(0x7f323c0f8620 msgr2=0x7f323c102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:01:47.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 shutdown_connections 2024-02-19T19:01:47.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:01:47.949+0000 7f324364e700 1 -- 172.21.15.77:0/3488271845 wait complete. 2024-02-19T19:01:48.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:48 smithi077 ceph-mon[29247]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:48.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:48 smithi077 ceph-mon[29247]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:48.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:48 smithi077 ceph-mon[29247]: mgrmap e25: smithi104.sdpzmi(active, since 73s), standbys: smithi077.yiiqpd 2024-02-19T19:01:48.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:48 smithi077 ceph-mon[29247]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:48.248 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:48 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3804838392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:01:48.423 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:48 smithi104 ceph-mon[42320]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:48.424 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:48 smithi104 ceph-mon[42320]: from='client.14722 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:48.424 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:48 smithi104 ceph-mon[42320]: mgrmap e25: smithi104.sdpzmi(active, since 73s), standbys: smithi077.yiiqpd 2024-02-19T19:01:48.424 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:48 smithi104 ceph-mon[42320]: from='client.14726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:48.424 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:48 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3804838392' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:01:49.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:49 smithi104 ceph-mon[42320]: pgmap v40: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:49.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:49 smithi104 ceph-mon[42320]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:49.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:49 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:49.443 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:49 smithi077 ceph-mon[29247]: pgmap v40: 65 pgs: 65 active+clean; 588 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:49.444 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:49 smithi077 ceph-mon[29247]: from='client.14734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:01:49.444 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:49 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:51.182 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:51.182 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr fail", "who": "smithi104.sdpzmi"}]: dispatch 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr fail", "who": "smithi104.sdpzmi"}]: dispatch 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: osdmap e48: 6 total, 6 up, 6 in 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd='[{"prefix": "mgr fail", "who": "smithi104.sdpzmi"}]': finished 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: mgrmap e26: smithi077.yiiqpd(active, starting, since 0.00824316s) 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.ovassg"}]: dispatch 2024-02-19T19:01:51.183 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.jmjalp"}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.rudtoq"}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.wpcqst"}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi077.yiiqpd", "id": "smithi077.yiiqpd"}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:01:51.184 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:01:51.185 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: Manager daemon smithi077.yiiqpd is now available 2024-02-19T19:01:51.185 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:51.185 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:51.185 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:51.185 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:01:51.185 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/trash_purge_schedule"}]: dispatch 2024-02-19T19:01:51.211 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:51.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:51.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:01:51.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:01:51.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' 2024-02-19T19:01:51.212 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 172.21.15.104:0/1302441304' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr fail", "who": "smithi104.sdpzmi"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "mgr fail", "who": "smithi104.sdpzmi"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: osdmap e48: 6 total, 6 up, 6 in 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.24355 ' entity='mgr.smithi104.sdpzmi' cmd='[{"prefix": "mgr fail", "who": "smithi104.sdpzmi"}]': finished 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: mgrmap e26: smithi077.yiiqpd(active, starting, since 0.00824316s) 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi077"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata", "id": "smithi104"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.ovassg"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi077.jmjalp"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.rudtoq"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi104.wpcqst"}]: dispatch 2024-02-19T19:01:51.213 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi077.yiiqpd", "id": "smithi077.yiiqpd"}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: Manager daemon smithi077.yiiqpd is now available 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/mirror_snapshot_schedule"}]: dispatch 2024-02-19T19:01:51.214 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi077.yiiqpd/trash_purge_schedule"}]: dispatch 2024-02-19T19:01:52.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:52 smithi104 ceph-mon[42320]: [19/Feb/2024:19:01:50] ENGINE Bus STARTING 2024-02-19T19:01:52.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:52 smithi104 ceph-mon[42320]: [19/Feb/2024:19:01:50] ENGINE Serving on https://172.21.15.77:7150 2024-02-19T19:01:52.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:52 smithi104 ceph-mon[42320]: [19/Feb/2024:19:01:50] ENGINE Serving on http://172.21.15.77:8765 2024-02-19T19:01:52.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:52 smithi104 ceph-mon[42320]: [19/Feb/2024:19:01:50] ENGINE Bus STARTED 2024-02-19T19:01:52.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:52 smithi104 ceph-mon[42320]: mgrmap e27: smithi077.yiiqpd(active, since 1.00783s) 2024-02-19T19:01:52.310 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:52 smithi104 ceph-mon[42320]: pgmap v3: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:52.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:52 smithi077 ceph-mon[29247]: [19/Feb/2024:19:01:50] ENGINE Bus STARTING 2024-02-19T19:01:52.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:52 smithi077 ceph-mon[29247]: [19/Feb/2024:19:01:50] ENGINE Serving on https://172.21.15.77:7150 2024-02-19T19:01:52.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:52 smithi077 ceph-mon[29247]: [19/Feb/2024:19:01:50] ENGINE Serving on http://172.21.15.77:8765 2024-02-19T19:01:52.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:52 smithi077 ceph-mon[29247]: [19/Feb/2024:19:01:50] ENGINE Bus STARTED 2024-02-19T19:01:52.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:52 smithi077 ceph-mon[29247]: mgrmap e27: smithi077.yiiqpd(active, since 1.00783s) 2024-02-19T19:01:52.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:52 smithi077 ceph-mon[29247]: pgmap v3: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:53.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:53 smithi077 ceph-mon[29247]: pgmap v4: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:53.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:53 smithi104 ceph-mon[42320]: pgmap v4: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:54.028 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:54 smithi104 ceph-mon[42320]: mgrmap e28: smithi077.yiiqpd(active, since 3s) 2024-02-19T19:01:54.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:54 smithi077 ceph-mon[29247]: mgrmap e28: smithi077.yiiqpd(active, since 3s) 2024-02-19T19:01:54.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:54 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:54.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:54 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:54.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:54 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:54.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:54 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:55.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:55 smithi104 ceph-mon[42320]: pgmap v5: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:55.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:55 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:55.182 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:55 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:55.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:55 smithi077 ceph-mon[29247]: pgmap v5: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:55.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:55 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:55.260 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:55 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:57.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:56 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:57.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:56 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:57.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:56 smithi077 ceph-mon[29247]: pgmap v6: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:57.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:56 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:57.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:56 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:57.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:56 smithi104 ceph-mon[42320]: pgmap v6: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:01:58.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:57 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:58.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:57 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:58.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:57 smithi077 ceph-mon[29247]: Standby manager daemon smithi104.sdpzmi started 2024-02-19T19:01:58.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:57 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/4117659999' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/crt"}]: dispatch 2024-02-19T19:01:58.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:57 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/4117659999' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:01:58.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:57 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/4117659999' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/key"}]: dispatch 2024-02-19T19:01:58.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:57 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/4117659999' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:01:58.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:57 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:58.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:57 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:01:58.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:57 smithi104 ceph-mon[42320]: Standby manager daemon smithi104.sdpzmi started 2024-02-19T19:01:58.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:57 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/4117659999' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/crt"}]: dispatch 2024-02-19T19:01:58.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:57 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/4117659999' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:01:58.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:57 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/4117659999' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/key"}]: dispatch 2024-02-19T19:01:58.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:57 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/4117659999' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:01:59.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:58 smithi077 ceph-mon[29247]: mgrmap e29: smithi077.yiiqpd(active, since 7s), standbys: smithi104.sdpzmi 2024-02-19T19:01:59.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:58 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi104.sdpzmi", "id": "smithi104.sdpzmi"}]: dispatch 2024-02-19T19:01:59.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:01:58 smithi077 ceph-mon[29247]: pgmap v7: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-19T19:01:59.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:58 smithi104 ceph-mon[42320]: mgrmap e29: smithi077.yiiqpd(active, since 7s), standbys: smithi104.sdpzmi 2024-02-19T19:01:59.063 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:58 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr metadata", "who": "smithi104.sdpzmi", "id": "smithi104.sdpzmi"}]: dispatch 2024-02-19T19:01:59.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:01:58 smithi104 ceph-mon[42320]: pgmap v7: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-19T19:02:00.403 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:00 smithi077 ceph-mon[29247]: pgmap v8: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:02:00.563 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:00 smithi104 ceph-mon[42320]: pgmap v8: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-19T19:02:02.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:02.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:02.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: Updating smithi077:/etc/ceph/ceph.conf 2024-02-19T19:02:02.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: Updating smithi104:/etc/ceph/ceph.conf 2024-02-19T19:02:02.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:01 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: Updating smithi077:/etc/ceph/ceph.conf 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: Updating smithi104:/etc/ceph/ceph.conf 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:02.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:01 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:03.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:02 smithi077 ceph-mon[29247]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:02:03.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:02 smithi077 ceph-mon[29247]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:02:03.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:02 smithi077 ceph-mon[29247]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:02:03.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:02 smithi077 ceph-mon[29247]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:02:03.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:02 smithi077 ceph-mon[29247]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:02:03.014 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:02 smithi077 ceph-mon[29247]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:02:03.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:02 smithi077 ceph-mon[29247]: Reconfiguring prometheus.smithi077 (dependencies changed)... 2024-02-19T19:02:03.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:02 smithi077 ceph-mon[29247]: pgmap v9: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-19T19:02:03.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:02 smithi104 ceph-mon[42320]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:02:03.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:02 smithi104 ceph-mon[42320]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.conf 2024-02-19T19:02:03.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:02 smithi104 ceph-mon[42320]: Updating smithi077:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:02:03.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:02 smithi104 ceph-mon[42320]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-02-19T19:02:03.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:02 smithi104 ceph-mon[42320]: Updating smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:02:03.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:02 smithi104 ceph-mon[42320]: Updating smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/config/ceph.client.admin.keyring 2024-02-19T19:02:03.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:02 smithi104 ceph-mon[42320]: Reconfiguring prometheus.smithi077 (dependencies changed)... 2024-02-19T19:02:03.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:02 smithi104 ceph-mon[42320]: pgmap v9: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-19T19:02:04.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:03 smithi077 ceph-mon[29247]: Reconfiguring daemon prometheus.smithi077 on smithi077 2024-02-19T19:02:04.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:03 smithi104 ceph-mon[42320]: Reconfiguring daemon prometheus.smithi077 on smithi077 2024-02-19T19:02:05.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:04 smithi077 ceph-mon[29247]: pgmap v10: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:02:05.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:04 smithi104 ceph-mon[42320]: pgmap v10: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:02:06.181 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:05 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:06.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:05 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:07.144 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:06 smithi077 ceph-mon[29247]: pgmap v11: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:02:07.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:06 smithi104 ceph-mon[42320]: pgmap v11: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:02:08.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:08 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:08.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:08 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:08.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:08 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:02:08.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:08 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:02:08.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:08 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:08.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:08 smithi077 ceph-mon[29247]: pgmap v12: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:02:08.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:08 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:08.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:08 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:08.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:08 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:02:08.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:08 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:02:08.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:08 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:08.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:08 smithi104 ceph-mon[42320]: pgmap v12: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-19T19:02:10.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:09 smithi077 ceph-mon[29247]: Upgrade: Updating mgr.smithi104.sdpzmi 2024-02-19T19:02:10.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:09 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.sdpzmi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:02:10.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:09 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:02:10.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:09 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:10.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:09 smithi077 ceph-mon[29247]: Deploying daemon mgr.smithi104.sdpzmi on smithi104 2024-02-19T19:02:10.313 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:09 smithi104 ceph-mon[42320]: Upgrade: Updating mgr.smithi104.sdpzmi 2024-02-19T19:02:10.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:09 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi104.sdpzmi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-19T19:02:10.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:09 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-19T19:02:10.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:09 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:10.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:09 smithi104 ceph-mon[42320]: Deploying daemon mgr.smithi104.sdpzmi on smithi104 2024-02-19T19:02:11.220 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:10 smithi104 ceph-mon[42320]: pgmap v13: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:11.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:10 smithi077 ceph-mon[29247]: pgmap v13: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:12.285 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:12 smithi104 ceph-mon[42320]: pgmap v14: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:12.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:12 smithi077 ceph-mon[29247]: pgmap v14: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:14.045 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:13 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:14.045 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:13 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:14.045 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:13 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:14.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:13 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:14.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:13 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:14.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:13 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:15.187 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:14 smithi104 ceph-mon[42320]: pgmap v15: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:15.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:14 smithi077 ceph-mon[29247]: pgmap v15: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:16.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:16 smithi104 ceph-mon[42320]: pgmap v16: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:16.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:16 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:16.426 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:16 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:16.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:16 smithi077 ceph-mon[29247]: pgmap v16: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:16.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:16 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:16.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:16 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:18.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.055+0000 7f9823c2d700 1 -- 172.21.15.77:0/4236324896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 msgr2=0x7f981c0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.055+0000 7f9823c2d700 1 --2- 172.21.15.77:0/4236324896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 0x7f981c0fe550 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f9818004940 tx=0x7f9818009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:18.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.055+0000 7f9823c2d700 1 -- 172.21.15.77:0/4236324896 shutdown_connections 2024-02-19T19:02:18.075 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.055+0000 7f9823c2d700 1 --2- 172.21.15.77:0/4236324896 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 0x7f981c0fe550 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.055+0000 7f9823c2d700 1 --2- 172.21.15.77:0/4236324896 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f981c0fcef0 0x7f981c0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.055+0000 7f9823c2d700 1 -- 172.21.15.77:0/4236324896 >> 172.21.15.77:0/4236324896 conn(0x7f981c0f8620 msgr2=0x7f981c0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:18.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.055+0000 7f9823c2d700 1 -- 172.21.15.77:0/4236324896 shutdown_connections 2024-02-19T19:02:18.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.055+0000 7f9823c2d700 1 -- 172.21.15.77:0/4236324896 wait complete. 2024-02-19T19:02:18.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.056+0000 7f9823c2d700 1 Processor -- start 2024-02-19T19:02:18.076 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.056+0000 7f9823c2d700 1 -- start start 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.056+0000 7f9823c2d700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f981c0fcef0 0x7f981c10d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.056+0000 7f9823c2d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 0x7f981c10de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.056+0000 7f9823c2d700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c10e430 con 0x7f981c0fe0e0 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.056+0000 7f9823c2d700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981c10ecd0 con 0x7f981c0fcef0 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.056+0000 7f98211c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 0x7f981c10de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98211c8700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 0x7f981c10de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51824/0 (socket says 172.21.15.77:51824) 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98211c8700 1 -- 172.21.15.77:0/1354720294 learned_addr learned my addr 172.21.15.77:0/1354720294 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98219c9700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f981c0fcef0 0x7f981c10d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98219c9700 1 -- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 msgr2=0x7f981c10de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98219c9700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 0x7f981c10de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.078 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98219c9700 1 -- 172.21.15.77:0/1354720294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9818005040 con 0x7f981c0fcef0 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98211c8700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 0x7f981c10de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98219c9700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f981c0fcef0 0x7f981c10d8f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f981c0fcc30 tx=0x7f980c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98127fc700 1 -- 172.21.15.77:0/1354720294 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f980c004e90 con 0x7f981c0fcef0 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981c10ef50 con 0x7f981c0fcef0 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98127fc700 1 -- 172.21.15.77:0/1354720294 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f980c00f040 con 0x7f981c0fcef0 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f98127fc700 1 -- 172.21.15.77:0/1354720294 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f980c00c980 con 0x7f981c0fcef0 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.057+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f981c10f490 con 0x7f981c0fcef0 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.058+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981c04bed0 con 0x7f981c0fcef0 2024-02-19T19:02:18.079 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.059+0000 7f98127fc700 1 -- 172.21.15.77:0/1354720294 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 29) v1 ==== 91309+0+0 (secure 0 0 0) 0x7f980c0090d0 con 0x7f981c0fcef0 2024-02-19T19:02:18.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.059+0000 7f98127fc700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f980806d200 0x7f980806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.059+0000 7f98127fc700 1 -- 172.21.15.77:0/1354720294 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f980c08c140 con 0x7f981c0fcef0 2024-02-19T19:02:18.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.059+0000 7f98211c8700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f980806d200 0x7f980806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.080 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.060+0000 7f98211c8700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f980806d200 0x7f980806f6b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f981800b0d0 tx=0x7f9818009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:18.083 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.064+0000 7f98127fc700 1 -- 172.21.15.77:0/1354720294 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f980c059b80 con 0x7f981c0fcef0 2024-02-19T19:02:18.275 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:18 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:18.275 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:18 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:18.275 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:18 smithi104 ceph-mon[42320]: pgmap v17: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:18.319 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.299+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f981c10f930 con 0x7f980806d200 2024-02-19T19:02:18.320 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.301+0000 7f98127fc700 1 -- 172.21.15.77:0/1354720294 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f981c10f930 con 0x7f980806d200 2024-02-19T19:02:18.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f980806d200 msgr2=0x7f980806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f980806d200 0x7f980806f6b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f981800b0d0 tx=0x7f9818009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:18.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f981c0fcef0 msgr2=0x7f981c10d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.322 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f981c0fcef0 0x7f981c10d8f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f981c0fcc30 tx=0x7f980c00b040 comp rx=0 tx=0).stop 2024-02-19T19:02:18.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 shutdown_connections 2024-02-19T19:02:18.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f980806d200 0x7f980806f6b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f981c0fcef0 0x7f981c10d8f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 --2- 172.21.15.77:0/1354720294 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f981c0fe0e0 0x7f981c10de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 >> 172.21.15.77:0/1354720294 conn(0x7f981c0f8620 msgr2=0x7f981c1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:18.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 shutdown_connections 2024-02-19T19:02:18.323 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.303+0000 7f9823c2d700 1 -- 172.21.15.77:0/1354720294 wait complete. 2024-02-19T19:02:18.332 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T19:02:18.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:18 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:18.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:18 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:18.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:18 smithi077 ceph-mon[29247]: pgmap v17: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:18.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.405+0000 7f5b5ae01700 1 -- 172.21.15.77:0/3576669761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fcfb0 msgr2=0x7f5b540fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.405+0000 7f5b5ae01700 1 --2- 172.21.15.77:0/3576669761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fcfb0 0x7f5b540fd420 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f5b44004940 tx=0x7f5b44009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:18.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.405+0000 7f5b5ae01700 1 -- 172.21.15.77:0/3576669761 shutdown_connections 2024-02-19T19:02:18.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.405+0000 7f5b5ae01700 1 --2- 172.21.15.77:0/3576669761 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fcfb0 0x7f5b540fd420 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.405+0000 7f5b5ae01700 1 --2- 172.21.15.77:0/3576669761 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b540fbe40 0x7f5b540fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.425 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.405+0000 7f5b5ae01700 1 -- 172.21.15.77:0/3576669761 >> 172.21.15.77:0/3576669761 conn(0x7f5b540f74e0 msgr2=0x7f5b540f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:18.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.405+0000 7f5b5ae01700 1 -- 172.21.15.77:0/3576669761 shutdown_connections 2024-02-19T19:02:18.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.405+0000 7f5b5ae01700 1 -- 172.21.15.77:0/3576669761 wait complete. 2024-02-19T19:02:18.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.406+0000 7f5b5ae01700 1 Processor -- start 2024-02-19T19:02:18.426 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.406+0000 7f5b5ae01700 1 -- start start 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.406+0000 7f5b5ae01700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fbe40 0x7f5b5410c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.406+0000 7f5b5ae01700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b540fcfb0 0x7f5b5410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.406+0000 7f5b5ae01700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5410d310 con 0x7f5b540fbe40 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.406+0000 7f5b5ae01700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b5410dbb0 con 0x7f5b540fcfb0 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b58b9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fbe40 0x7f5b5410c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b58b9d700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fbe40 0x7f5b5410c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51840/0 (socket says 172.21.15.77:51840) 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b58b9d700 1 -- 172.21.15.77:0/1790956146 learned_addr learned my addr 172.21.15.77:0/1790956146 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b4bfff700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b540fcfb0 0x7f5b5410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b58b9d700 1 -- 172.21.15.77:0/1790956146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b540fcfb0 msgr2=0x7f5b5410cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.428 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b58b9d700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b540fcfb0 0x7f5b5410cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b58b9d700 1 -- 172.21.15.77:0/1790956146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b44005040 con 0x7f5b540fbe40 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b58b9d700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fbe40 0x7f5b5410c7d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000fd10 tx=0x7f5b500046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b497fa700 1 -- 172.21.15.77:0/1790956146 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b5000dcf0 con 0x7f5b540fbe40 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b5410de30 con 0x7f5b540fbe40 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b497fa700 1 -- 172.21.15.77:0/1790956146 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5b5000de50 con 0x7f5b540fbe40 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b497fa700 1 -- 172.21.15.77:0/1790956146 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5b500126a0 con 0x7f5b540fbe40 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.407+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b5410e370 con 0x7f5b540fbe40 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.408+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b5404bed0 con 0x7f5b540fbe40 2024-02-19T19:02:18.429 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.409+0000 7f5b497fa700 1 -- 172.21.15.77:0/1790956146 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 29) v1 ==== 91309+0+0 (secure 0 0 0) 0x7f5b50007ab0 con 0x7f5b540fbe40 2024-02-19T19:02:18.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.409+0000 7f5b497fa700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f5b3c06d200 0x7f5b3c06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.409+0000 7f5b497fa700 1 -- 172.21.15.77:0/1790956146 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f5b5008ecf0 con 0x7f5b540fbe40 2024-02-19T19:02:18.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.409+0000 7f5b4bfff700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f5b3c06d200 0x7f5b3c06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.430 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.410+0000 7f5b4bfff700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f5b3c06d200 0x7f5b3c06f6b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5b44004f90 tx=0x7f5b44009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:18.433 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.414+0000 7f5b497fa700 1 -- 172.21.15.77:0/1790956146 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f5b5005c6a0 con 0x7f5b540fbe40 2024-02-19T19:02:18.648 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.629+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b5410e810 con 0x7f5b3c06d200 2024-02-19T19:02:18.649 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.630+0000 7f5b497fa700 1 -- 172.21.15.77:0/1790956146 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f5b5410e810 con 0x7f5b3c06d200 2024-02-19T19:02:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.632+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f5b3c06d200 msgr2=0x7f5b3c06f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.632+0000 7f5b5ae01700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f5b3c06d200 0x7f5b3c06f6b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5b44004f90 tx=0x7f5b44009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.632+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fbe40 msgr2=0x7f5b5410c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.632+0000 7f5b5ae01700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fbe40 0x7f5b5410c7d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5b5000fd10 tx=0x7f5b500046b0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.633+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 shutdown_connections 2024-02-19T19:02:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.633+0000 7f5b5ae01700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f5b3c06d200 0x7f5b3c06f6b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.633+0000 7f5b5ae01700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f5b540fbe40 0x7f5b5410c7d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.652 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.633+0000 7f5b5ae01700 1 --2- 172.21.15.77:0/1790956146 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f5b540fcfb0 0x7f5b5410cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.633+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 >> 172.21.15.77:0/1790956146 conn(0x7f5b540f74e0 msgr2=0x7f5b54100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.633+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 shutdown_connections 2024-02-19T19:02:18.653 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.633+0000 7f5b5ae01700 1 -- 172.21.15.77:0/1790956146 wait complete. 2024-02-19T19:02:18.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/1373690788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 msgr2=0x7ff3f40fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 --2- 172.21.15.77:0/1373690788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 0x7ff3f40fc250 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009ce0 tx=0x7ff3f0004d10 comp rx=0 tx=0).stop 2024-02-19T19:02:18.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/1373690788 shutdown_connections 2024-02-19T19:02:18.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 --2- 172.21.15.77:0/1373690788 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f40fcfb0 0x7ff3f40fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.757 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 --2- 172.21.15.77:0/1373690788 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 0x7ff3f40fc250 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/1373690788 >> 172.21.15.77:0/1373690788 conn(0x7ff3f40f74e0 msgr2=0x7ff3f40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:18.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/1373690788 shutdown_connections 2024-02-19T19:02:18.758 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/1373690788 wait complete. 2024-02-19T19:02:18.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 Processor -- start 2024-02-19T19:02:18.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.738+0000 7ff3fb9f4700 1 -- start start 2024-02-19T19:02:18.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3fb9f4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 0x7ff3f4112ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3fb9f4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f40fcfb0 0x7ff3f4115000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3fb9f4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4066350 con 0x7ff3f40fbe40 2024-02-19T19:02:18.759 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3fb9f4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f4066490 con 0x7ff3f40fcfb0 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3f9790700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 0x7ff3f4112ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3f9790700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 0x7ff3f4112ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51860/0 (socket says 172.21.15.77:51860) 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3f9790700 1 -- 172.21.15.77:0/3666793462 learned_addr learned my addr 172.21.15.77:0/3666793462 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3f8f8f700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f40fcfb0 0x7ff3f4115000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3f9790700 1 -- 172.21.15.77:0/3666793462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f40fcfb0 msgr2=0x7ff3f4115000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3f9790700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f40fcfb0 0x7ff3f4115000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3f9790700 1 -- 172.21.15.77:0/3666793462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f0009940 con 0x7ff3f40fbe40 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3f9790700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 0x7ff3f4112ab0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009380 tx=0x7ff3f0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3e9ffb700 1 -- 172.21.15.77:0/3666793462 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f000e040 con 0x7ff3f40fbe40 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3e9ffb700 1 -- 172.21.15.77:0/3666793462 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3f001cd00 con 0x7ff3f40fbe40 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f4115540 con 0x7ff3f40fbe40 2024-02-19T19:02:18.760 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.739+0000 7ff3e9ffb700 1 -- 172.21.15.77:0/3666793462 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3f0012b70 con 0x7ff3f40fbe40 2024-02-19T19:02:18.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.740+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f41158e0 con 0x7ff3f40fbe40 2024-02-19T19:02:18.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.740+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f404bed0 con 0x7ff3f40fbe40 2024-02-19T19:02:18.761 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.741+0000 7ff3e9ffb700 1 -- 172.21.15.77:0/3666793462 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 29) v1 ==== 91309+0+0 (secure 0 0 0) 0x7ff3f001c830 con 0x7ff3f40fbe40 2024-02-19T19:02:18.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.741+0000 7ff3e9ffb700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff3e006d200 0x7ff3e006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:18.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.742+0000 7ff3e9ffb700 1 -- 172.21.15.77:0/3666793462 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff3f008ebd0 con 0x7ff3f40fbe40 2024-02-19T19:02:18.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.742+0000 7ff3f8f8f700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff3e006d200 0x7ff3e006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:18.762 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.742+0000 7ff3f8f8f700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff3e006d200 0x7ff3e006f6b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009b80 tx=0x7ff3e400b040 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:18.765 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.746+0000 7ff3e9ffb700 1 -- 172.21.15.77:0/3666793462 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7ff3f0058a50 con 0x7ff3f40fbe40 2024-02-19T19:02:18.969 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.950+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3f4101990 con 0x7ff3e006d200 2024-02-19T19:02:18.980 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.961+0000 7ff3e9ffb700 1 -- 172.21.15.77:0/3666793462 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3498 (secure 0 0 0) 0x7ff3f4101990 con 0x7ff3e006d200 2024-02-19T19:02:18.980 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:02:18.980 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (81s) 0s ago 10m 18.8M - 0.25.0 c8568f914cd2 46ae75dc6421 2024-02-19T19:02:18.980 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (11m) 0s ago 11m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T19:02:18.980 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (8m) 2s ago 8m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (58s) 0s ago 9m 80.1M - 9.4.7 954c08fa6188 454ae3504841 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (3m) 0s ago 3m 14.0M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (3m) 0s ago 3m 27.6M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (3m) 2s ago 3m 13.9M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (3m) 2s ago 3m 12.9M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:8443,9283,8765 running (40s) 0s ago 12m 491M - 19.0.0-1450-gf78a58c0 f081f847070a e536fa0dfb57 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283,8765 running (6s) 2s ago 8m 66.7M - 19.0.0-1450-gf78a58c0 f081f847070a 8af90712a2d0 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (12m) 0s ago 12m 56.3M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (8m) 2s ago 8m 41.2M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (87s) 0s ago 11m 16.4M - 1.5.0 0da6a335fe13 e8ff8e3e80d6 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (50s) 2s ago 8m 8652k - 1.5.0 0da6a335fe13 8230c18c1dd0 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (7m) 0s ago 7m 47.4M 1557M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (7m) 0s ago 7m 46.5M 1557M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T19:02:18.981 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (6m) 0s ago 6m 46.4M 1557M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T19:02:18.982 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (6m) 2s ago 6m 46.0M 2581M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T19:02:18.982 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (5m) 2s ago 5m 46.7M 2581M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T19:02:18.982 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (5m) 2s ago 5m 47.1M 2581M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T19:02:18.982 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (12s) 0s ago 8m 36.5M - 2.43.0 a07b618ecd1d 4c8401993e4a 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.963+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff3e006d200 msgr2=0x7ff3e006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.963+0000 7ff3fb9f4700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff3e006d200 0x7ff3e006f6b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff3e4009b80 tx=0x7ff3e400b040 comp rx=0 tx=0).stop 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.963+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 msgr2=0x7ff3f4112ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.963+0000 7ff3fb9f4700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 0x7ff3f4112ab0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7ff3f0009380 tx=0x7ff3f0004d10 comp rx=0 tx=0).stop 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.963+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 shutdown_connections 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.963+0000 7ff3fb9f4700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff3e006d200 0x7ff3e006f6b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.963+0000 7ff3fb9f4700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff3f40fbe40 0x7ff3f4112ab0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.963+0000 7ff3fb9f4700 1 --2- 172.21.15.77:0/3666793462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff3f40fcfb0 0x7ff3f4115000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.964+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 >> 172.21.15.77:0/3666793462 conn(0x7ff3f40f74e0 msgr2=0x7ff3f4100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.964+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 shutdown_connections 2024-02-19T19:02:18.983 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:18.964+0000 7ff3fb9f4700 1 -- 172.21.15.77:0/3666793462 wait complete. 2024-02-19T19:02:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.063+0000 7ff584a3a700 1 -- 172.21.15.77:0/562200225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 msgr2=0x7ff5800fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.063+0000 7ff584a3a700 1 --2- 172.21.15.77:0/562200225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 0x7ff5800fd420 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff574004940 tx=0x7ff574009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 -- 172.21.15.77:0/562200225 shutdown_connections 2024-02-19T19:02:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 --2- 172.21.15.77:0/562200225 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 0x7ff5800fd420 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.084 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 --2- 172.21.15.77:0/562200225 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5800fbe40 0x7ff5800fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 -- 172.21.15.77:0/562200225 >> 172.21.15.77:0/562200225 conn(0x7ff5800f7520 msgr2=0x7ff5800f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 -- 172.21.15.77:0/562200225 shutdown_connections 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 -- 172.21.15.77:0/562200225 wait complete. 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 Processor -- start 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 -- start start 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5800fbe40 0x7ff5800fff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 0x7ff5800fe380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800fe980 con 0x7ff5800fcfb0 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff584a3a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800feaf0 con 0x7ff5800fbe40 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff57dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 0x7ff5800fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff57dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 0x7ff5800fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51868/0 (socket says 172.21.15.77:51868) 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff57dd9b700 1 -- 172.21.15.77:0/3898037895 learned_addr learned my addr 172.21.15.77:0/3898037895 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.064+0000 7ff57e59c700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5800fbe40 0x7ff5800fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:19.085 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff57dd9b700 1 -- 172.21.15.77:0/3898037895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5800fbe40 msgr2=0x7ff5800fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff57dd9b700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5800fbe40 0x7ff5800fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff57dd9b700 1 -- 172.21.15.77:0/3898037895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574005040 con 0x7ff5800fcfb0 2024-02-19T19:02:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff57dd9b700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 0x7ff5800fe380 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff5740093b0 tx=0x7ff574009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff56effd700 1 -- 172.21.15.77:0/3898037895 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff57400e040 con 0x7ff5800fcfb0 2024-02-19T19:02:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5800fed70 con 0x7ff5800fcfb0 2024-02-19T19:02:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff56effd700 1 -- 172.21.15.77:0/3898037895 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff57400bcc0 con 0x7ff5800fcfb0 2024-02-19T19:02:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff56effd700 1 -- 172.21.15.77:0/3898037895 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff57401c630 con 0x7ff5800fcfb0 2024-02-19T19:02:19.086 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.065+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff580115970 con 0x7ff5800fcfb0 2024-02-19T19:02:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.066+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58004bed0 con 0x7ff5800fcfb0 2024-02-19T19:02:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.067+0000 7ff56effd700 1 -- 172.21.15.77:0/3898037895 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 29) v1 ==== 91309+0+0 (secure 0 0 0) 0x7ff574013980 con 0x7ff5800fcfb0 2024-02-19T19:02:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.067+0000 7ff56effd700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff56806d200 0x7ff56806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.067+0000 7ff56effd700 1 -- 172.21.15.77:0/3898037895 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7ff57408db90 con 0x7ff5800fcfb0 2024-02-19T19:02:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.067+0000 7ff57e59c700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff56806d200 0x7ff56806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:19.087 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.067+0000 7ff57e59c700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff56806d200 0x7ff56806f6b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff5800ffc90 tx=0x7ff57000b040 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:19.091 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.071+0000 7ff56effd700 1 -- 172.21.15.77:0/3898037895 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7ff57405b5c0 con 0x7ff5800fcfb0 2024-02-19T19:02:19.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.354+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7ff580104930 con 0x7ff5800fcfb0 2024-02-19T19:02:19.373 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.354+0000 7ff56effd700 1 -- 172.21.15.77:0/3898037895 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7ff57405b150 con 0x7ff5800fcfb0 2024-02-19T19:02:19.373 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 12, 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:02:19.374 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T19:02:19.375 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:02:19.375 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff56806d200 msgr2=0x7ff56806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff56806d200 0x7ff56806f6b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff5800ffc90 tx=0x7ff57000b040 comp rx=0 tx=0).stop 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 msgr2=0x7ff5800fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 0x7ff5800fe380 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff5740093b0 tx=0x7ff574009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 shutdown_connections 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7ff56806d200 0x7ff56806f6b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7ff5800fbe40 0x7ff5800fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 --2- 172.21.15.77:0/3898037895 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7ff5800fcfb0 0x7ff5800fe380 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 >> 172.21.15.77:0/3898037895 conn(0x7ff5800f7520 msgr2=0x7ff580065f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 shutdown_connections 2024-02-19T19:02:19.376 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.356+0000 7ff584a3a700 1 -- 172.21.15.77:0/3898037895 wait complete. 2024-02-19T19:02:19.478 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 -- 172.21.15.77:0/2075541183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 msgr2=0x7fd8040fd290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 --2- 172.21.15.77:0/2075541183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 0x7fd8040fd290 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec004940 tx=0x7fd7ec009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 -- 172.21.15.77:0/2075541183 shutdown_connections 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 --2- 172.21.15.77:0/2075541183 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8040fe080 0x7fd8040fe4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 --2- 172.21.15.77:0/2075541183 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 0x7fd8040fd290 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 -- 172.21.15.77:0/2075541183 >> 172.21.15.77:0/2075541183 conn(0x7fd8040f8590 msgr2=0x7fd8040fa9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 -- 172.21.15.77:0/2075541183 shutdown_connections 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 -- 172.21.15.77:0/2075541183 wait complete. 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.459+0000 7fd80b0e0700 1 Processor -- start 2024-02-19T19:02:19.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd80b0e0700 1 -- start start 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd80b0e0700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 0x7fd80410d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd80b0e0700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8040fe080 0x7fd80410dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd80b0e0700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80410e340 con 0x7fd8040fce80 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd80b0e0700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80410ebe0 con 0x7fd8040fe080 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd808e7c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 0x7fd80410d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd808e7c700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 0x7fd80410d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:51888/0 (socket says 172.21.15.77:51888) 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd808e7c700 1 -- 172.21.15.77:0/3766089752 learned_addr learned my addr 172.21.15.77:0/3766089752 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd803fff700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8040fe080 0x7fd80410dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd808e7c700 1 -- 172.21.15.77:0/3766089752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8040fe080 msgr2=0x7fd80410dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd808e7c700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8040fe080 0x7fd80410dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd808e7c700 1 -- 172.21.15.77:0/3766089752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec005040 con 0x7fd8040fce80 2024-02-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.460+0000 7fd808e7c700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 0x7fd80410d800 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec000c00 tx=0x7fd7ec009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.461+0000 7fd8017fa700 1 -- 172.21.15.77:0/3766089752 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ec021070 con 0x7fd8040fce80 2024-02-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.461+0000 7fd8017fa700 1 -- 172.21.15.77:0/3766089752 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7ec009430 con 0x7fd8040fce80 2024-02-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.461+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd80410ee60 con 0x7fd8040fce80 2024-02-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.461+0000 7fd8017fa700 1 -- 172.21.15.77:0/3766089752 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ec00e070 con 0x7fd8040fce80 2024-02-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.461+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd80410f3a0 con 0x7fd8040fce80 2024-02-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.461+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80404bed0 con 0x7fd8040fce80 2024-02-19T19:02:19.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.462+0000 7fd8017fa700 1 -- 172.21.15.77:0/3766089752 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 29) v1 ==== 91309+0+0 (secure 0 0 0) 0x7fd7ec01c400 con 0x7fd8040fce80 2024-02-19T19:02:19.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.462+0000 7fd8017fa700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7f006d200 0x7fd7f006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:19.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.462+0000 7fd8017fa700 1 -- 172.21.15.77:0/3766089752 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd7ec08da10 con 0x7fd8040fce80 2024-02-19T19:02:19.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.462+0000 7fd803fff700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7f006d200 0x7fd7f006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:19.483 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.463+0000 7fd803fff700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7f006d200 0x7fd7f006f6b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4009b80 tx=0x7fd7f400b040 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:19.484 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:19 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:19.485 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:19 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:19.485 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:19 smithi104 ceph-mon[42320]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:19.485 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:19 smithi104 ceph-mon[42320]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:19.485 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:19 smithi104 ceph-mon[42320]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:19.486 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.467+0000 7fd8017fa700 1 -- 172.21.15.77:0/3766089752 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd7ec05b4e0 con 0x7fd8040fce80 2024-02-19T19:02:19.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:19 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:19.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:19 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:19.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:19 smithi077 ceph-mon[29247]: from='client.24475 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:19.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:19 smithi077 ceph-mon[29247]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:19.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:19 smithi077 ceph-mon[29247]: from='client.14776 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:19.703 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.684+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd80410eff0 con 0x7fd7f006d200 2024-02-19T19:02:19.704 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.685+0000 7fd8017fa700 1 -- 172.21.15.77:0/3766089752 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fd80410eff0 con 0x7fd7f006d200 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading mgr daemons", 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T19:02:19.705 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:02:19.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7f006d200 msgr2=0x7fd7f006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:19.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7f006d200 0x7fd7f006f6b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd7f4009b80 tx=0x7fd7f400b040 comp rx=0 tx=0).stop 2024-02-19T19:02:19.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 msgr2=0x7fd80410d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:19.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 0x7fd80410d800 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec000c00 tx=0x7fd7ec009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:19.707 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 shutdown_connections 2024-02-19T19:02:19.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7f006d200 0x7fd7f006f6b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd8040fce80 0x7fd80410d800 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 --2- 172.21.15.77:0/3766089752 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd8040fe080 0x7fd80410dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:19.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 >> 172.21.15.77:0/3766089752 conn(0x7fd8040f8590 msgr2=0x7fd804101340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:19.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 shutdown_connections 2024-02-19T19:02:19.708 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:19.688+0000 7fd80b0e0700 1 -- 172.21.15.77:0/3766089752 wait complete. 2024-02-19T19:02:20.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/3898037895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:20.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:20.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: Standby manager daemon smithi104.sdpzmi restarted 2024-02-19T19:02:20.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: Standby manager daemon smithi104.sdpzmi started 2024-02-19T19:02:20.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/2569096208' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/crt"}]: dispatch 2024-02-19T19:02:20.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/2569096208' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:02:20.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/2569096208' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/key"}]: dispatch 2024-02-19T19:02:20.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: from='mgr.? 172.21.15.104:0/2569096208' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:02:20.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: pgmap v18: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:20.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:20 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/3898037895' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: Standby manager daemon smithi104.sdpzmi restarted 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: Standby manager daemon smithi104.sdpzmi started 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/2569096208' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/crt"}]: dispatch 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/2569096208' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/2569096208' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi104.sdpzmi/key"}]: dispatch 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: from='mgr.? 172.21.15.104:0/2569096208' entity='mgr.smithi104.sdpzmi' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: pgmap v18: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:20.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:20 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:21.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: mgrmap e30: smithi077.yiiqpd(active, since 30s), standbys: smithi104.sdpzmi 2024-02-19T19:02:21.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:21.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi077.yiiqpd"}]: dispatch 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi077.yiiqpd"}]': finished 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.sdpzmi"}]: dispatch 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.sdpzmi"}]': finished 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:21 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.790 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: mgrmap e30: smithi077.yiiqpd(active, since 30s), standbys: smithi104.sdpzmi 2024-02-19T19:02:21.790 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.790 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.790 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi077.yiiqpd"}]: dispatch 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi077.yiiqpd"}]': finished 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.sdpzmi"}]: dispatch 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi104.sdpzmi"}]': finished 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:21.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:21 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: Upgrade: Setting container_image for all mgr 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: Upgrade: Setting container_image for all rgw 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: Upgrade: Setting container_image for all rbd-mirror 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: Upgrade: Setting container_image for all ceph-exporter 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: Upgrade: Setting container_image for all iscsi 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: Upgrade: Setting container_image for all nfs 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: Upgrade: Setting container_image for all nvmeof 2024-02-19T19:02:22.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:22 smithi077 ceph-mon[29247]: pgmap v19: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:22.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: Upgrade: Setting container_image for all mgr 2024-02-19T19:02:22.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: Upgrade: Setting container_image for all rgw 2024-02-19T19:02:22.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: Upgrade: Setting container_image for all rbd-mirror 2024-02-19T19:02:22.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: Upgrade: Setting container_image for all cephfs-mirror 2024-02-19T19:02:22.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: Upgrade: Setting container_image for all ceph-exporter 2024-02-19T19:02:22.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: Upgrade: Setting container_image for all iscsi 2024-02-19T19:02:22.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: Upgrade: Setting container_image for all nfs 2024-02-19T19:02:22.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: Upgrade: Setting container_image for all nvmeof 2024-02-19T19:02:22.815 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:22 smithi104 ceph-mon[42320]: pgmap v19: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:24.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:23 smithi077 ceph-mon[29247]: Upgrade: Updating node-exporter.smithi077 (1/2) 2024-02-19T19:02:24.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:23 smithi077 ceph-mon[29247]: Deploying daemon node-exporter.smithi077 on smithi077 2024-02-19T19:02:24.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:23 smithi104 ceph-mon[42320]: Upgrade: Updating node-exporter.smithi077 (1/2) 2024-02-19T19:02:24.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:23 smithi104 ceph-mon[42320]: Deploying daemon node-exporter.smithi077 on smithi077 2024-02-19T19:02:25.060 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:24 smithi077 ceph-mon[29247]: pgmap v20: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:25.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:24 smithi104 ceph-mon[42320]: pgmap v20: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:26 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:26 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:27.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:26 smithi104 ceph-mon[42320]: pgmap v21: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:27.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:26 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:27.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:26 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:27.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:26 smithi077 ceph-mon[29247]: pgmap v21: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:29.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:28 smithi104 ceph-mon[42320]: Upgrade: Updating node-exporter.smithi104 (2/2) 2024-02-19T19:02:29.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:28 smithi104 ceph-mon[42320]: Deploying daemon node-exporter.smithi104 on smithi104 2024-02-19T19:02:29.064 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:28 smithi104 ceph-mon[42320]: pgmap v22: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:28 smithi077 ceph-mon[29247]: Upgrade: Updating node-exporter.smithi104 (2/2) 2024-02-19T19:02:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:28 smithi077 ceph-mon[29247]: Deploying daemon node-exporter.smithi104 on smithi104 2024-02-19T19:02:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:28 smithi077 ceph-mon[29247]: pgmap v22: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:30.396 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:30 smithi104 ceph-mon[42320]: pgmap v23: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:30.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:30 smithi077 ceph-mon[29247]: pgmap v23: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:31.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:31 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:31.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:31 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:31.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:31 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:31.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:31 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:31.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:31 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:31.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:31 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:32.670 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:32 smithi077 ceph-mon[29247]: pgmap v24: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:32.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:32 smithi104 ceph-mon[42320]: pgmap v24: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:34.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:34 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:34.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:34 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:34.748 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:34 smithi077 ceph-mon[29247]: pgmap v25: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:34 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:34 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:34.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:34 smithi104 ceph-mon[42320]: pgmap v25: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:35.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:35 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:35.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:35 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:35.745 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:35 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:35.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:35 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:35.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:35 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:35.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:35 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:36.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:36 smithi077 ceph-mon[29247]: pgmap v26: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:36.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:36 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:36.810 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:36 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:36.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:36 smithi104 ceph-mon[42320]: pgmap v26: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:36.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:36 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:36.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:36 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:38.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:38 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:38.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:38 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:38.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:38 smithi077 ceph-mon[29247]: pgmap v27: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:38.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:38 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:38.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:38 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:38.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:38 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:38.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:38 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:38.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:38 smithi104 ceph-mon[42320]: pgmap v27: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:38.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:38 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:38.814 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:38 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:40.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:40 smithi077 ceph-mon[29247]: pgmap v28: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:40.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:40 smithi104 ceph-mon[42320]: pgmap v28: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:42.367 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:42.367 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:42.367 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:42.367 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:42.367 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:42 smithi077 ceph-mon[29247]: pgmap v29: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:42.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:42.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:42.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:42 smithi104 ceph-mon[42320]: pgmap v29: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:44.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:44 smithi077 ceph-mon[29247]: Upgrade: Updating prometheus.smithi077 2024-02-19T19:02:44.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:44 smithi077 ceph-mon[29247]: Deploying daemon prometheus.smithi077 on smithi077 2024-02-19T19:02:44.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:44 smithi077 ceph-mon[29247]: pgmap v30: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:44.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:44 smithi104 ceph-mon[42320]: Upgrade: Updating prometheus.smithi077 2024-02-19T19:02:44.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:44 smithi104 ceph-mon[42320]: Deploying daemon prometheus.smithi077 on smithi077 2024-02-19T19:02:44.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:44 smithi104 ceph-mon[42320]: pgmap v30: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:46.270 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:46 smithi077 ceph-mon[29247]: pgmap v31: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:46.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:46 smithi104 ceph-mon[42320]: pgmap v31: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:49.219 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:48 smithi077 ceph-mon[29247]: pgmap v32: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:49.265 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:48 smithi104 ceph-mon[42320]: pgmap v32: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:49.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.816+0000 7fefa7ddb700 1 -- 172.21.15.77:0/2809733244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0107950 msgr2=0x7fefa0107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:49.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.816+0000 7fefa7ddb700 1 --2- 172.21.15.77:0/2809733244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0107950 0x7fefa0107d60 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fef9c004940 tx=0x7fef9c009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:49.837 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.816+0000 7fefa7ddb700 1 -- 172.21.15.77:0/2809733244 shutdown_connections 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.816+0000 7fefa7ddb700 1 --2- 172.21.15.77:0/2809733244 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa0108b60 0x7fefa0108fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.816+0000 7fefa7ddb700 1 --2- 172.21.15.77:0/2809733244 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0107950 0x7fefa0107d60 secure :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fef9c004940 tx=0x7fef9c009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.816+0000 7fefa7ddb700 1 -- 172.21.15.77:0/2809733244 >> 172.21.15.77:0/2809733244 conn(0x7fefa00672f0 msgr2=0x7fefa0067710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.816+0000 7fefa7ddb700 1 -- 172.21.15.77:0/2809733244 shutdown_connections 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.816+0000 7fefa7ddb700 1 -- 172.21.15.77:0/2809733244 wait complete. 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.817+0000 7fefa7ddb700 1 Processor -- start 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.817+0000 7fefa7ddb700 1 -- start start 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.817+0000 7fefa7ddb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0108b60 0x7fefa0069620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.817+0000 7fefa7ddb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa0069b60 0x7fefa006aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.817+0000 7fefa7ddb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa006a020 con 0x7fefa0108b60 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.817+0000 7fefa7ddb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefa006a190 con 0x7fefa0069b60 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.818+0000 7fefa5b77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0108b60 0x7fefa0069620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.818+0000 7fefa5b77700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0108b60 0x7fefa0069620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38448/0 (socket says 172.21.15.77:38448) 2024-02-19T19:02:49.838 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.818+0000 7fefa5b77700 1 -- 172.21.15.77:0/1151569651 learned_addr learned my addr 172.21.15.77:0/1151569651 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.818+0000 7fefa5376700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa0069b60 0x7fefa006aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.819+0000 7fefa5b77700 1 -- 172.21.15.77:0/1151569651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa0069b60 msgr2=0x7fefa006aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.819+0000 7fefa5b77700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa0069b60 0x7fefa006aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.819+0000 7fefa5b77700 1 -- 172.21.15.77:0/1151569651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90005040 con 0x7fefa0108b60 2024-02-19T19:02:49.839 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.819+0000 7fefa5b77700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0108b60 0x7fefa0069620 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0043d0 tx=0x7fef9c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:49.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.820+0000 7fef967fc700 1 -- 172.21.15.77:0/1151569651 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9c00e070 con 0x7fefa0108b60 2024-02-19T19:02:49.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.820+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c005040 con 0x7fefa0108b60 2024-02-19T19:02:49.840 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.820+0000 7fef967fc700 1 -- 172.21.15.77:0/1151569651 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef9c009430 con 0x7fefa0108b60 2024-02-19T19:02:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.821+0000 7fef967fc700 1 -- 172.21.15.77:0/1151569651 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fef9c01ce90 con 0x7fefa0108b60 2024-02-19T19:02:49.842 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.821+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefa006b3b0 con 0x7fefa0108b60 2024-02-19T19:02:49.843 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.823+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefa0112470 con 0x7fefa0108b60 2024-02-19T19:02:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.824+0000 7fef967fc700 1 -- 172.21.15.77:0/1151569651 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 30) v1 ==== 91309+0+0 (secure 0 0 0) 0x7fef9c01c400 con 0x7fefa0108b60 2024-02-19T19:02:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.825+0000 7fef967fc700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fef8c06d100 0x7fef8c06f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:49.845 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.825+0000 7fef967fc700 1 -- 172.21.15.77:0/1151569651 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fef9c08d090 con 0x7fefa0108b60 2024-02-19T19:02:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.827+0000 7fefa5376700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fef8c06d100 0x7fef8c06f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:49.847 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.828+0000 7fefa5376700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fef8c06d100 0x7fef8c06f5b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fefa006a690 tx=0x7fef90009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:49.852 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:49.832+0000 7fef967fc700 1 -- 172.21.15.77:0/1151569651 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fef9c092060 con 0x7fefa0108b60 2024-02-19T19:02:50.138 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.117+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fefa006b740 con 0x7fef8c06d100 2024-02-19T19:02:50.140 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.121+0000 7fef967fc700 1 -- 172.21.15.77:0/1151569651 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fefa006b740 con 0x7fef8c06d100 2024-02-19T19:02:50.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.124+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fef8c06d100 msgr2=0x7fef8c06f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:50.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.124+0000 7fefa7ddb700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fef8c06d100 0x7fef8c06f5b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fefa006a690 tx=0x7fef90009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:50.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.124+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0108b60 msgr2=0x7fefa0069620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:50.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.124+0000 7fefa7ddb700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0108b60 0x7fefa0069620 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0043d0 tx=0x7fef9c009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:50.144 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.125+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 shutdown_connections 2024-02-19T19:02:50.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.125+0000 7fefa7ddb700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fef8c06d100 0x7fef8c06f5b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.125+0000 7fefa7ddb700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fefa0108b60 0x7fefa0069620 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.125+0000 7fefa7ddb700 1 --2- 172.21.15.77:0/1151569651 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fefa0069b60 0x7fefa006aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.125+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 >> 172.21.15.77:0/1151569651 conn(0x7fefa00672f0 msgr2=0x7fefa0106730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:50.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.125+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 shutdown_connections 2024-02-19T19:02:50.145 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.125+0000 7fefa7ddb700 1 -- 172.21.15.77:0/1151569651 wait complete. 2024-02-19T19:02:50.159 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T19:02:50.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.237+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/653595919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fcfb0 msgr2=0x7fd9ac0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:50.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.237+0000 7fd9b1ea3700 1 --2- 172.21.15.77:0/653595919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fcfb0 0x7fd9ac0fd420 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0004940 tx=0x7fd9a0009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:50.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.238+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/653595919 shutdown_connections 2024-02-19T19:02:50.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.238+0000 7fd9b1ea3700 1 --2- 172.21.15.77:0/653595919 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fcfb0 0x7fd9ac0fd420 secure :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fd9a0004940 tx=0x7fd9a0009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:50.257 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.238+0000 7fd9b1ea3700 1 --2- 172.21.15.77:0/653595919 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9ac0fbe40 0x7fd9ac0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.238+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/653595919 >> 172.21.15.77:0/653595919 conn(0x7fd9ac0f74e0 msgr2=0x7fd9ac0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:50.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.238+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/653595919 shutdown_connections 2024-02-19T19:02:50.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.238+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/653595919 wait complete. 2024-02-19T19:02:50.258 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.239+0000 7fd9b1ea3700 1 Processor -- start 2024-02-19T19:02:50.259 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.239+0000 7fd9b1ea3700 1 -- start start 2024-02-19T19:02:50.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9b1ea3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fbe40 0x7fd9ac10da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:50.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9b1ea3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9ac10dfb0 0x7fd9ac10ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:50.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9b1ea3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac10e470 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9b1ea3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ac10e5b0 con 0x7fd9ac10dfb0 2024-02-19T19:02:50.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9aaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9ac10dfb0 0x7fd9ac10ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:50.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9ab7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fbe40 0x7fd9ac10da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:50.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9ab7fe700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fbe40 0x7fd9ac10da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38456/0 (socket says 172.21.15.77:38456) 2024-02-19T19:02:50.260 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9aaffd700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9ac10dfb0 0x7fd9ac10ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.77:38030/0 (socket says 172.21.15.77:38030) 2024-02-19T19:02:50.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.240+0000 7fd9ab7fe700 1 -- 172.21.15.77:0/870376462 learned_addr learned my addr 172.21.15.77:0/870376462 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:50.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd9ab7fe700 1 -- 172.21.15.77:0/870376462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9ac10dfb0 msgr2=0x7fd9ac10ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:50.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd9ab7fe700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9ac10dfb0 0x7fd9ac10ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd9ab7fe700 1 -- 172.21.15.77:0/870376462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a0005040 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd9ab7fe700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fbe40 0x7fd9ac10da70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00eb40 tx=0x7fd99c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:50.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd993fff700 1 -- 172.21.15.77:0/870376462 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd99c007cc0 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd993fff700 1 -- 172.21.15.77:0/870376462 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd99c00f040 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.261 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd993fff700 1 -- 172.21.15.77:0/870376462 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd99c00ca10 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ac10f450 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.241+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ac10f8a0 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.243+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ac04bed0 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.262 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.243+0000 7fd993fff700 1 -- 172.21.15.77:0/870376462 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 30) v1 ==== 91309+0+0 (secure 0 0 0) 0x7fd99c00cb70 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.244+0000 7fd993fff700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd99406d200 0x7fd99406f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:50.263 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.244+0000 7fd993fff700 1 -- 172.21.15.77:0/870376462 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd99c08c2d0 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.244+0000 7fd9aaffd700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd99406d200 0x7fd99406f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:50.264 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.245+0000 7fd9aaffd700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd99406d200 0x7fd99406f6b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b0d0 tx=0x7fd9a0009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:50.268 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.249+0000 7fd993fff700 1 -- 172.21.15.77:0/870376462 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd99c059db0 con 0x7fd9ac0fbe40 2024-02-19T19:02:50.489 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.470+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9ac101990 con 0x7fd99406d200 2024-02-19T19:02:50.491 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.471+0000 7fd993fff700 1 -- 172.21.15.77:0/870376462 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7fd9ac101990 con 0x7fd99406d200 2024-02-19T19:02:50.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd99406d200 msgr2=0x7fd99406f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:50.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd99406d200 0x7fd99406f6b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd9a000b0d0 tx=0x7fd9a0009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:50.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fbe40 msgr2=0x7fd9ac10da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:50.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fbe40 0x7fd9ac10da70 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fd99c00eb40 tx=0x7fd99c00b040 comp rx=0 tx=0).stop 2024-02-19T19:02:50.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 shutdown_connections 2024-02-19T19:02:50.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd99406d200 0x7fd99406f6b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd9ac0fbe40 0x7fd9ac10da70 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.495 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 --2- 172.21.15.77:0/870376462 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd9ac10dfb0 0x7fd9ac10ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.475+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 >> 172.21.15.77:0/870376462 conn(0x7fd9ac0f74e0 msgr2=0x7fd9ac100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:50.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.476+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 shutdown_connections 2024-02-19T19:02:50.496 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.476+0000 7fd9b1ea3700 1 -- 172.21.15.77:0/870376462 wait complete. 2024-02-19T19:02:50.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.636+0000 7f52cf7fe700 1 -- 172.21.15.77:0/634314072 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52c4013b70 con 0x7f52d00fb2c0 2024-02-19T19:02:50.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.637+0000 7f52d7108700 1 -- 172.21.15.77:0/634314072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d00fb2c0 msgr2=0x7f52d00fb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:50.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.637+0000 7f52d7108700 1 --2- 172.21.15.77:0/634314072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d00fb2c0 0x7f52d00fb6d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f52c4004940 tx=0x7f52c4009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:50.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.637+0000 7f52d7108700 1 -- 172.21.15.77:0/634314072 shutdown_connections 2024-02-19T19:02:50.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.637+0000 7f52d7108700 1 --2- 172.21.15.77:0/634314072 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52d00fbca0 0x7f52d00fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.637+0000 7f52d7108700 1 --2- 172.21.15.77:0/634314072 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d00fb2c0 0x7f52d00fb6d0 secure :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f52c4004940 tx=0x7f52c4009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:50.657 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.637+0000 7f52d7108700 1 -- 172.21.15.77:0/634314072 >> 172.21.15.77:0/634314072 conn(0x7f52d00f6ca0 msgr2=0x7f52d00f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:50.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.639+0000 7f52d7108700 1 -- 172.21.15.77:0/634314072 shutdown_connections 2024-02-19T19:02:50.658 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.639+0000 7f52d7108700 1 -- 172.21.15.77:0/634314072 wait complete. 2024-02-19T19:02:50.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.640+0000 7f52d7108700 1 Processor -- start 2024-02-19T19:02:50.659 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.640+0000 7f52d7108700 1 -- start start 2024-02-19T19:02:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.640+0000 7f52d7108700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d010fee0 0x7f52d01102f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.640+0000 7f52d7108700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52d0110830 0x7f52d0111300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.640+0000 7f52d7108700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0111950 con 0x7f52d010fee0 2024-02-19T19:02:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.640+0000 7f52d7108700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0111ac0 con 0x7f52d0110830 2024-02-19T19:02:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52cffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52d0110830 0x7f52d0111300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52cffff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52d0110830 0x7f52d0111300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.104:3300/0 says I am v2:172.21.15.77:38056/0 (socket says 172.21.15.77:38056) 2024-02-19T19:02:50.661 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52cffff700 1 -- 172.21.15.77:0/1198363570 learned_addr learned my addr 172.21.15.77:0/1198363570 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52d4ea4700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d010fee0 0x7f52d01102f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52cffff700 1 -- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d010fee0 msgr2=0x7f52d01102f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52cffff700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d010fee0 0x7f52d01102f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52cffff700 1 -- 172.21.15.77:0/1198363570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c4005040 con 0x7f52d0110830 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52cffff700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52d0110830 0x7f52d0111300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00eb40 tx=0x7f52bc00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.641+0000 7f52cd7fa700 1 -- 172.21.15.77:0/1198363570 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52bc007cc0 con 0x7f52d0110830 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.642+0000 7f52cd7fa700 1 -- 172.21.15.77:0/1198363570 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52bc00f040 con 0x7f52d0110830 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.642+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d0111d40 con 0x7f52d0110830 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.642+0000 7f52cd7fa700 1 -- 172.21.15.77:0/1198363570 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52bc00ca00 con 0x7f52d0110830 2024-02-19T19:02:50.662 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.642+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d0116540 con 0x7f52d0110830 2024-02-19T19:02:50.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.643+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b40052e0 con 0x7f52d0110830 2024-02-19T19:02:50.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.643+0000 7f52cd7fa700 1 -- 172.21.15.77:0/1198363570 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 30) v1 ==== 91309+0+0 (secure 0 0 0) 0x7f52bc00cb60 con 0x7f52d0110830 2024-02-19T19:02:50.663 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.644+0000 7f52cd7fa700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f52c006d200 0x7f52c006f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:50.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.644+0000 7f52cd7fa700 1 -- 172.21.15.77:0/1198363570 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f52bc08c220 con 0x7f52d0110830 2024-02-19T19:02:50.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.644+0000 7f52d4ea4700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f52c006d200 0x7f52c006f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:50.664 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.645+0000 7f52d4ea4700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f52c006d200 0x7f52c006f6b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52c4004890 tx=0x7f52c4009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:50.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.652+0000 7f52cd7fa700 1 -- 172.21.15.77:0/1198363570 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f52bc059d70 con 0x7f52d0110830 2024-02-19T19:02:51.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.986+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f52b4000bf0 con 0x7f52c006d200 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (113s) 14s ago 11m 25.8M - 0.25.0 c8568f914cd2 46ae75dc6421 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (11m) 14s ago 11m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (9m) 17s ago 9m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (90s) 14s ago 9m 80.1M - 9.4.7 954c08fa6188 454ae3504841 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (4m) 14s ago 4m 14.1M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (4m) 14s ago 4m 27.5M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (4m) 17s ago 4m 14.0M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (4m) 17s ago 4m 13.0M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T19:02:51.016 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:8443,9283,8765 running (72s) 14s ago 12m 494M - 19.0.0-1450-gf78a58c0 f081f847070a e536fa0dfb57 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283,8765 running (38s) 17s ago 8m 449M - 19.0.0-1450-gf78a58c0 f081f847070a 8af90712a2d0 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (12m) 14s ago 13m 56.2M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (8m) 17s ago 8m 42.0M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (25s) 14s ago 11m 8056k - 1.5.0 0da6a335fe13 9a8b0cdc4900 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (21s) 17s ago 9m 3409k - 1.5.0 0da6a335fe13 87a965a5c174 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (8m) 14s ago 8m 47.5M 1557M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (7m) 14s ago 7m 46.6M 1557M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (7m) 14s ago 7m 46.5M 1557M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (6m) 17s ago 6m 46.2M 2581M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (6m) 17s ago 6m 46.6M 2581M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (5m) 17s ago 5m 47.1M 2581M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 starting - - - - 2024-02-19T19:02:51.017 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.996+0000 7f52cd7fa700 1 -- 172.21.15.77:0/1198363570 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f52b4000bf0 con 0x7f52c006d200 2024-02-19T19:02:51.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.998+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f52c006d200 msgr2=0x7f52c006f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.998+0000 7f52d7108700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f52c006d200 0x7f52c006f6b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f52c4004890 tx=0x7f52c4009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:51.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.998+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52d0110830 msgr2=0x7f52d0111300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.018 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.998+0000 7f52d7108700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52d0110830 0x7f52d0111300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f52bc00eb40 tx=0x7f52bc00b040 comp rx=0 tx=0).stop 2024-02-19T19:02:51.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.999+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 shutdown_connections 2024-02-19T19:02:51.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.999+0000 7f52d7108700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f52c006d200 0x7f52c006f6b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.999+0000 7f52d7108700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f52d010fee0 0x7f52d01102f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.999+0000 7f52d7108700 1 --2- 172.21.15.77:0/1198363570 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f52d0110830 0x7f52d0111300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.999+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 >> 172.21.15.77:0/1198363570 conn(0x7f52d00f6ca0 msgr2=0x7f52d00f87b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:51.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.999+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 shutdown_connections 2024-02-19T19:02:51.019 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:50.999+0000 7f52d7108700 1 -- 172.21.15.77:0/1198363570 wait complete. 2024-02-19T19:02:51.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.108+0000 7f1933fc9700 1 -- 172.21.15.77:0/153988984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fd3f0 msgr2=0x7f192c0fd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.128 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.108+0000 7f1933fc9700 1 --2- 172.21.15.77:0/153988984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fd3f0 0x7f192c0fd800 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1928004940 tx=0x7f1928009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:51.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.108+0000 7f1933fc9700 1 -- 172.21.15.77:0/153988984 shutdown_connections 2024-02-19T19:02:51.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.108+0000 7f1933fc9700 1 --2- 172.21.15.77:0/153988984 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f192c0fc140 0x7f192c0fc590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.109+0000 7f1933fc9700 1 --2- 172.21.15.77:0/153988984 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fd3f0 0x7f192c0fd800 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.109+0000 7f1933fc9700 1 -- 172.21.15.77:0/153988984 >> 172.21.15.77:0/153988984 conn(0x7f192c0f6ca0 msgr2=0x7f192c0f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:51.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.109+0000 7f1933fc9700 1 -- 172.21.15.77:0/153988984 shutdown_connections 2024-02-19T19:02:51.129 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.109+0000 7f1933fc9700 1 -- 172.21.15.77:0/153988984 wait complete. 2024-02-19T19:02:51.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1933fc9700 1 Processor -- start 2024-02-19T19:02:51.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1933fc9700 1 -- start start 2024-02-19T19:02:51.130 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1933fc9700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fc140 0x7f192c10c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1933fc9700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f192c0fd3f0 0x7f192c10cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1933fc9700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c10d3d0 con 0x7f192c0fc140 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1933fc9700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192c10dc70 con 0x7f192c0fd3f0 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1931d65700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fc140 0x7f192c10c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1931564700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f192c0fd3f0 0x7f192c10cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1931d65700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fc140 0x7f192c10c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38500/0 (socket says 172.21.15.77:38500) 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.110+0000 7f1931d65700 1 -- 172.21.15.77:0/861105100 learned_addr learned my addr 172.21.15.77:0/861105100 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.111+0000 7f1931564700 1 -- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fc140 msgr2=0x7f192c10c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.111+0000 7f1931564700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fc140 0x7f192c10c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.111+0000 7f1931564700 1 -- 172.21.15.77:0/861105100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1928005040 con 0x7f192c0fd3f0 2024-02-19T19:02:51.131 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.111+0000 7f1931564700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f192c0fd3f0 0x7f192c10cdd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f191c009fd0 tx=0x7f191c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:51.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.112+0000 7f19227fc700 1 -- 172.21.15.77:0/861105100 <== mon.1 v2:172.21.15.104:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f191c00ed30 con 0x7f192c0fd3f0 2024-02-19T19:02:51.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.112+0000 7f19227fc700 1 -- 172.21.15.77:0/861105100 <== mon.1 v2:172.21.15.104:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f191c00ee90 con 0x7f192c0fd3f0 2024-02-19T19:02:51.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.112+0000 7f19227fc700 1 -- 172.21.15.77:0/861105100 <== mon.1 v2:172.21.15.104:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f191c012670 con 0x7f192c0fd3f0 2024-02-19T19:02:51.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.112+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192c10def0 con 0x7f192c0fd3f0 2024-02-19T19:02:51.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.112+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f192c10e430 con 0x7f192c0fd3f0 2024-02-19T19:02:51.132 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.113+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f192c04bed0 con 0x7f192c0fd3f0 2024-02-19T19:02:51.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.115+0000 7f19227fc700 1 -- 172.21.15.77:0/861105100 <== mon.1 v2:172.21.15.104:3300/0 4 ==== mgrmap(e 30) v1 ==== 91309+0+0 (secure 0 0 0) 0x7f191c018030 con 0x7f192c0fd3f0 2024-02-19T19:02:51.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.115+0000 7f19227fc700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f191806d200 0x7f191806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:51.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.115+0000 7f1931d65700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f191806d200 0x7f191806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:51.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.115+0000 7f19227fc700 1 -- 172.21.15.77:0/861105100 <== mon.1 v2:172.21.15.104:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f191c016070 con 0x7f192c0fd3f0 2024-02-19T19:02:51.135 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.116+0000 7f1931d65700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f191806d200 0x7f191806f6b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1928009fd0 tx=0x7f1928009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:51.141 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.122+0000 7f19227fc700 1 -- 172.21.15.77:0/861105100 <== mon.1 v2:172.21.15.104:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f191c05c980 con 0x7f192c0fd3f0 2024-02-19T19:02:51.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:51.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:51.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:51.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:50 smithi077 ceph-mon[29247]: pgmap v33: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:51.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:50 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:51.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:51.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:51.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:02:51.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:50 smithi104 ceph-mon[42320]: pgmap v33: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:51.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:50 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:02:51.436 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.417+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f192c104930 con 0x7f192c0fd3f0 2024-02-19T19:02:51.437 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.418+0000 7f19227fc700 1 -- 172.21.15.77:0/861105100 <== mon.1 v2:172.21.15.104:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f191c05c510 con 0x7f192c0fd3f0 2024-02-19T19:02:51.437 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:02:51.437 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T19:02:51.437 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:02:51.437 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 12, 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T19:02:51.438 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:02:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f191806d200 msgr2=0x7f191806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f191806d200 0x7f191806f6b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1928009fd0 tx=0x7f1928009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:51.440 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f192c0fd3f0 msgr2=0x7f192c10cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f192c0fd3f0 0x7f192c10cdd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f191c009fd0 tx=0x7f191c00b040 comp rx=0 tx=0).stop 2024-02-19T19:02:51.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 shutdown_connections 2024-02-19T19:02:51.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f191806d200 0x7f191806f6b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f192c0fc140 0x7f192c10c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 --2- 172.21.15.77:0/861105100 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f192c0fd3f0 0x7f192c10cdd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 >> 172.21.15.77:0/861105100 conn(0x7f192c0f6ca0 msgr2=0x7f192c100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:51.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 shutdown_connections 2024-02-19T19:02:51.441 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.421+0000 7f1933fc9700 1 -- 172.21.15.77:0/861105100 wait complete. 2024-02-19T19:02:51.547 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.527+0000 7f6e34ade700 1 -- 172.21.15.77:0/1951850048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 msgr2=0x7f6e30102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.527+0000 7f6e34ade700 1 --2- 172.21.15.77:0/1951850048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 0x7f6e30102520 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f6e24004940 tx=0x7f6e24009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.527+0000 7f6e34ade700 1 -- 172.21.15.77:0/1951850048 shutdown_connections 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.527+0000 7f6e34ade700 1 --2- 172.21.15.77:0/1951850048 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 0x7f6e30102520 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.527+0000 7f6e34ade700 1 --2- 172.21.15.77:0/1951850048 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300663e0 0x7f6e300ffaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.527+0000 7f6e34ade700 1 -- 172.21.15.77:0/1951850048 >> 172.21.15.77:0/1951850048 conn(0x7f6e300f74e0 msgr2=0x7f6e300f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.527+0000 7f6e34ade700 1 -- 172.21.15.77:0/1951850048 shutdown_connections 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.527+0000 7f6e34ade700 1 -- 172.21.15.77:0/1951850048 wait complete. 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e34ade700 1 Processor -- start 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e34ade700 1 -- start start 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e34ade700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300663e0 0x7f6e3010d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e34ade700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 0x7f6e3010de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e34ade700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3010e410 con 0x7f6e301000c0 2024-02-19T19:02:51.548 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e2dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 0x7f6e3010de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:51.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e34ade700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3010ecb0 con 0x7f6e300663e0 2024-02-19T19:02:51.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e2dd9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 0x7f6e3010de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:38518/0 (socket says 172.21.15.77:38518) 2024-02-19T19:02:51.549 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.528+0000 7f6e2dd9b700 1 -- 172.21.15.77:0/3901884506 learned_addr learned my addr 172.21.15.77:0/3901884506 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:02:51.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e2e59c700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300663e0 0x7f6e3010d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:51.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e2dd9b700 1 -- 172.21.15.77:0/3901884506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300663e0 msgr2=0x7f6e3010d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e2dd9b700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300663e0 0x7f6e3010d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e2dd9b700 1 -- 172.21.15.77:0/3901884506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e24005040 con 0x7f6e301000c0 2024-02-19T19:02:51.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e2dd9b700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 0x7f6e3010de10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6e24000c00 tx=0x7f6e24009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:51.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e1effd700 1 -- 172.21.15.77:0/3901884506 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e2400e040 con 0x7f6e301000c0 2024-02-19T19:02:51.550 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e1effd700 1 -- 172.21.15.77:0/3901884506 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e2400bcc0 con 0x7f6e301000c0 2024-02-19T19:02:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e1effd700 1 -- 172.21.15.77:0/3901884506 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6e2401c630 con 0x7f6e301000c0 2024-02-19T19:02:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.529+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e3010ef30 con 0x7f6e301000c0 2024-02-19T19:02:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.530+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e3010f3f0 con 0x7f6e301000c0 2024-02-19T19:02:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.531+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e3004bed0 con 0x7f6e301000c0 2024-02-19T19:02:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.531+0000 7f6e1effd700 1 -- 172.21.15.77:0/3901884506 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 30) v1 ==== 91309+0+0 (secure 0 0 0) 0x7f6e24013980 con 0x7f6e301000c0 2024-02-19T19:02:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.532+0000 7f6e1effd700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f6e1806d200 0x7f6e1806f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:02:51.551 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.532+0000 7f6e1effd700 1 -- 172.21.15.77:0/3901884506 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f6e2408d8d0 con 0x7f6e301000c0 2024-02-19T19:02:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.532+0000 7f6e2e59c700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f6e1806d200 0x7f6e1806f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:02:51.552 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.533+0000 7f6e2e59c700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f6e1806d200 0x7f6e1806f6b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009b80 tx=0x7f6e2000b040 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:02:51.559 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.540+0000 7f6e1effd700 1 -- 172.21.15.77:0/3901884506 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f6e2405b300 con 0x7f6e301000c0 2024-02-19T19:02:51.827 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.808+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e30100d60 con 0x7f6e1806d200 2024-02-19T19:02:51.831 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.809+0000 7f6e1effd700 1 -- 172.21.15.77:0/3901884506 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+367 (secure 0 0 0) 0x7f6e30100d60 con 0x7f6e1806d200 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading prometheus daemons", 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T19:02:51.832 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f6e1806d200 msgr2=0x7f6e1806f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f6e1806d200 0x7f6e1806f6b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6e20009b80 tx=0x7f6e2000b040 comp rx=0 tx=0).stop 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 msgr2=0x7f6e3010de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 0x7f6e3010de10 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f6e24000c00 tx=0x7f6e24009e30 comp rx=0 tx=0).stop 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 shutdown_connections 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f6e1806d200 0x7f6e1806f6b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f6e300663e0 0x7f6e3010d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 --2- 172.21.15.77:0/3901884506 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f6e301000c0 0x7f6e3010de10 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 >> 172.21.15.77:0/3901884506 conn(0x7f6e300f74e0 msgr2=0x7f6e300f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.811+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 shutdown_connections 2024-02-19T19:02:51.833 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:02:51.812+0000 7f6e34ade700 1 -- 172.21.15.77:0/3901884506 wait complete. 2024-02-19T19:02:52.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:51 smithi077 ceph-mon[29247]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:52.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:51 smithi077 ceph-mon[29247]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:52.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:51 smithi077 ceph-mon[29247]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:52.179 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:51 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/861105100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:52.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:51 smithi104 ceph-mon[42320]: from='client.14788 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:52.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:51 smithi104 ceph-mon[42320]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:52.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:51 smithi104 ceph-mon[42320]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:52.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:51 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/861105100' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:02:53.100 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:52 smithi077 ceph-mon[29247]: from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:53.100 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:52 smithi077 ceph-mon[29247]: pgmap v34: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:53.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:52 smithi104 ceph-mon[42320]: from='client.14804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:02:53.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:52 smithi104 ceph-mon[42320]: pgmap v34: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:54.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:54 smithi077 ceph-mon[29247]: pgmap v35: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:54.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:54 smithi104 ceph-mon[42320]: pgmap v35: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:57.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:56 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:57.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:56 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:57.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:56 smithi077 ceph-mon[29247]: pgmap v36: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:57.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:56 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:57.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:56 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:57.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:56 smithi104 ceph-mon[42320]: pgmap v36: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:58.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:58 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:58.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:58 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:58.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:02:58 smithi077 ceph-mon[29247]: pgmap v37: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:02:58.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:58 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:58.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:58 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:02:58.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:02:58 smithi104 ceph-mon[42320]: pgmap v37: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:00.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:00 smithi104 ceph-mon[42320]: pgmap v38: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:00.459 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:00 smithi077 ceph-mon[29247]: pgmap v38: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:02.444 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.446 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.446 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.446 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.446 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.446 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.446 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:02 smithi077 ceph-mon[29247]: pgmap v39: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:02.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:02.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:02.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:02 smithi104 ceph-mon[42320]: pgmap v39: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:04.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:04 smithi077 ceph-mon[29247]: Upgrade: Updating alertmanager.smithi077 2024-02-19T19:03:04.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:04 smithi077 ceph-mon[29247]: Deploying daemon alertmanager.smithi077 on smithi077 2024-02-19T19:03:04.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:04 smithi077 ceph-mon[29247]: pgmap v40: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:04.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:04 smithi104 ceph-mon[42320]: Upgrade: Updating alertmanager.smithi077 2024-02-19T19:03:04.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:04 smithi104 ceph-mon[42320]: Deploying daemon alertmanager.smithi077 on smithi077 2024-02-19T19:03:04.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:04 smithi104 ceph-mon[42320]: pgmap v40: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:05.384 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:05 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:05.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:05 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:06.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:06 smithi077 ceph-mon[29247]: pgmap v41: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:06.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:06 smithi104 ceph-mon[42320]: pgmap v41: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr:Note: switching to '1ce7ea8877c709fa862eac608550374bd96e166e'. 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr:state without impacting any branches by switching back to a branch. 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-02-19T19:03:07.645 INFO:tasks.workunit.client.0.smithi077.stderr: git switch -c 2024-02-19T19:03:07.646 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-02-19T19:03:07.646 INFO:tasks.workunit.client.0.smithi077.stderr:Or undo this operation with: 2024-02-19T19:03:07.646 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-02-19T19:03:07.646 INFO:tasks.workunit.client.0.smithi077.stderr: git switch - 2024-02-19T19:03:07.646 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-02-19T19:03:07.646 INFO:tasks.workunit.client.0.smithi077.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-19T19:03:07.646 INFO:tasks.workunit.client.0.smithi077.stderr: 2024-02-19T19:03:07.646 INFO:tasks.workunit.client.0.smithi077.stderr:HEAD is now at 1ce7ea8877c fix 2024-02-19T19:03:07.649 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-19T19:03:07.739 INFO:tasks.workunit.client.0.smithi077.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-19T19:03:07.741 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-19T19:03:07.741 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr:Note: switching to '1ce7ea8877c709fa862eac608550374bd96e166e'. 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr: 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr:state without impacting any branches by switching back to a branch. 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr: 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr: 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr: git switch -c 2024-02-19T19:03:08.862 INFO:tasks.workunit.client.1.smithi104.stderr: 2024-02-19T19:03:08.863 INFO:tasks.workunit.client.1.smithi104.stderr:Or undo this operation with: 2024-02-19T19:03:08.863 INFO:tasks.workunit.client.1.smithi104.stderr: 2024-02-19T19:03:08.863 INFO:tasks.workunit.client.1.smithi104.stderr: git switch - 2024-02-19T19:03:08.863 INFO:tasks.workunit.client.1.smithi104.stderr: 2024-02-19T19:03:08.863 INFO:tasks.workunit.client.1.smithi104.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-19T19:03:08.863 INFO:tasks.workunit.client.1.smithi104.stderr: 2024-02-19T19:03:08.863 INFO:tasks.workunit.client.1.smithi104.stderr:HEAD is now at 1ce7ea8877c fix 2024-02-19T19:03:08.867 DEBUG:teuthology.orchestra.run.smithi104:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-02-19T19:03:08.931 INFO:tasks.workunit.client.1.smithi104.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-19T19:03:08.933 INFO:tasks.workunit.client.1.smithi104.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-19T19:03:08.933 INFO:tasks.workunit.client.1.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-19T19:03:09.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:08 smithi077 ceph-mon[29247]: pgmap v42: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:09.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:08 smithi104 ceph-mon[42320]: pgmap v42: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:09.334 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-19T19:03:09.420 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-19T19:03:09.462 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-19T19:03:09.463 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-19T19:03:09.463 INFO:tasks.workunit.client.0.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-19T19:03:09.505 INFO:tasks.workunit.client.0.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-19T19:03:09.508 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T19:03:09.508 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-19T19:03:09.524 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-02-19T19:03:09.526 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-19T19:03:09.526 DEBUG:teuthology.orchestra.run.smithi077:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-02-19T19:03:09.584 INFO:tasks.workunit.client.0.smithi077.stderr:+ mkdir -p fsstress 2024-02-19T19:03:09.586 INFO:tasks.workunit.client.0.smithi077.stderr:+ pushd fsstress 2024-02-19T19:03:09.586 INFO:tasks.workunit.client.0.smithi077.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-19T19:03:09.586 INFO:tasks.workunit.client.0.smithi077.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-19T19:03:09.658 INFO:tasks.workunit.client.1.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-19T19:03:09.713 INFO:tasks.workunit.client.1.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-19T19:03:09.753 INFO:tasks.workunit.client.1.smithi104.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-02-19T19:03:09.755 INFO:tasks.workunit.client.1.smithi104.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-19T19:03:09.755 INFO:tasks.workunit.client.1.smithi104.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-19T19:03:09.794 INFO:tasks.workunit.client.1.smithi104.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-02-19T19:03:09.797 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T19:03:09.797 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-02-19T19:03:09.851 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-02-19T19:03:09.852 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-02-19T19:03:09.853 DEBUG:teuthology.orchestra.run.smithi104:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ce7ea8877c709fa862eac608550374bd96e166e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-02-19T19:03:09.912 INFO:tasks.workunit.client.1.smithi104.stderr:+ mkdir -p fsstress 2024-02-19T19:03:09.914 INFO:tasks.workunit.client.1.smithi104.stderr:+ pushd fsstress 2024-02-19T19:03:09.914 INFO:tasks.workunit.client.1.smithi104.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-19T19:03:09.914 INFO:tasks.workunit.client.1.smithi104.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-02-19T19:03:10.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:10 smithi077 ceph-mon[29247]: pgmap v43: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:10.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:10 smithi104 ceph-mon[42320]: pgmap v43: 65 pgs: 65 active+clean; 589 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-02-19T19:03:11.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:11 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:11.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:11 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:11.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:11 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:11.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:11 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:11.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:11 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:11.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:11 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:12.166 INFO:tasks.workunit.client.1.smithi104.stderr:+ tar xzf ltp-full.tgz 2024-02-19T19:03:12.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:12 smithi077 ceph-mon[29247]: pgmap v44: 65 pgs: 65 active+clean; 609 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s wr, 0 op/s 2024-02-19T19:03:12.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:12 smithi104 ceph-mon[42320]: pgmap v44: 65 pgs: 65 active+clean; 609 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s wr, 0 op/s 2024-02-19T19:03:13.385 INFO:tasks.workunit.client.0.smithi077.stderr:+ tar xzf ltp-full.tgz 2024-02-19T19:03:14.472 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:14 smithi077 ceph-mon[29247]: pgmap v45: 65 pgs: 65 active+clean; 609 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s wr, 0 op/s 2024-02-19T19:03:14.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:14 smithi104 ceph-mon[42320]: pgmap v45: 65 pgs: 65 active+clean; 609 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.9 KiB/s wr, 0 op/s 2024-02-19T19:03:17.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:16 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:17.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:16 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:17.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:16 smithi077 ceph-mon[29247]: pgmap v46: 65 pgs: 65 active+clean; 41 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s wr, 289 op/s 2024-02-19T19:03:17.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:16 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:17.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:16 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:17.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:16 smithi104 ceph-mon[42320]: pgmap v46: 65 pgs: 65 active+clean; 41 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s wr, 289 op/s 2024-02-19T19:03:18.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:18 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:18.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:18 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:18.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:18 smithi077 ceph-mon[29247]: pgmap v47: 65 pgs: 65 active+clean; 41 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s wr, 289 op/s 2024-02-19T19:03:18.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:18 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:18.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:18 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:18.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:18 smithi104 ceph-mon[42320]: pgmap v47: 65 pgs: 65 active+clean; 41 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s wr, 289 op/s 2024-02-19T19:03:20.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:20 smithi077 ceph-mon[29247]: pgmap v48: 65 pgs: 65 active+clean; 41 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s wr, 289 op/s 2024-02-19T19:03:20.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:20 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:20.389 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:20 smithi104 ceph-mon[42320]: pgmap v48: 65 pgs: 65 active+clean; 41 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 3.5 MiB/s wr, 289 op/s 2024-02-19T19:03:20.389 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:20 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:21.946 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.926+0000 7fadd7213700 1 -- 172.21.15.77:0/763875648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 msgr2=0x7fadd00ffaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.926+0000 7fadd7213700 1 --2- 172.21.15.77:0/763875648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 0x7fadd00ffaf0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fadc4004940 tx=0x7fadc4009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.926+0000 7fadd7213700 1 -- 172.21.15.77:0/763875648 shutdown_connections 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.926+0000 7fadd7213700 1 --2- 172.21.15.77:0/763875648 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd01000c0 0x7fadd0102520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.926+0000 7fadd7213700 1 --2- 172.21.15.77:0/763875648 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 0x7fadd00ffaf0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.926+0000 7fadd7213700 1 -- 172.21.15.77:0/763875648 >> 172.21.15.77:0/763875648 conn(0x7fadd00f7520 msgr2=0x7fadd00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.926+0000 7fadd7213700 1 -- 172.21.15.77:0/763875648 shutdown_connections 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.926+0000 7fadd7213700 1 -- 172.21.15.77:0/763875648 wait complete. 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.927+0000 7fadd7213700 1 Processor -- start 2024-02-19T19:03:21.947 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.927+0000 7fadd7213700 1 -- start start 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd7213700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 0x7fadd00fd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd7213700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd01000c0 0x7fadd00fb9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd7213700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00fbff0 con 0x7fadd0066c70 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd7213700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00fc160 con 0x7fadd01000c0 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd4faf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 0x7fadd00fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd4faf700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 0x7fadd00fd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46168/0 (socket says 172.21.15.77:46168) 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd4faf700 1 -- 172.21.15.77:0/2842263095 learned_addr learned my addr 172.21.15.77:0/2842263095 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadcffff700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd01000c0 0x7fadd00fb9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd4faf700 1 -- 172.21.15.77:0/2842263095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd01000c0 msgr2=0x7fadd00fb9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd4faf700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd01000c0 0x7fadd00fb9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd4faf700 1 -- 172.21.15.77:0/2842263095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc4005040 con 0x7fadd0066c70 2024-02-19T19:03:21.949 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadcffff700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd01000c0 0x7fadd00fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-19T19:03:21.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.928+0000 7fadd4faf700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 0x7fadd00fd570 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fadc40093b0 tx=0x7fadc4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:21.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.929+0000 7fadcd7fa700 1 -- 172.21.15.77:0/2842263095 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc400e040 con 0x7fadd0066c70 2024-02-19T19:03:21.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.929+0000 7fadcd7fa700 1 -- 172.21.15.77:0/2842263095 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fadc400bcc0 con 0x7fadd0066c70 2024-02-19T19:03:21.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.929+0000 7fadcd7fa700 1 -- 172.21.15.77:0/2842263095 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fadc401c630 con 0x7fadd0066c70 2024-02-19T19:03:21.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.929+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd00fc3e0 con 0x7fadd0066c70 2024-02-19T19:03:21.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.929+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd01159a0 con 0x7fadd0066c70 2024-02-19T19:03:21.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.930+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd0107a70 con 0x7fadd0066c70 2024-02-19T19:03:21.950 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.931+0000 7fadcd7fa700 1 -- 172.21.15.77:0/2842263095 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7fadc4013980 con 0x7fadd0066c70 2024-02-19T19:03:21.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.931+0000 7fadcd7fa700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fadc006d0e0 0x7fadc006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:21.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.932+0000 7fadcd7fa700 1 -- 172.21.15.77:0/2842263095 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fadc408d7b0 con 0x7fadd0066c70 2024-02-19T19:03:21.951 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.932+0000 7fadcffff700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fadc006d0e0 0x7fadc006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:21.952 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.932+0000 7fadcffff700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fadc006d0e0 0x7fadc006f590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fadd00669b0 tx=0x7fadbc00b040 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:21.960 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:21.941+0000 7fadcd7fa700 1 -- 172.21.15.77:0/2842263095 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fadc405b370 con 0x7fadd0066c70 2024-02-19T19:03:22.275 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.255+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fadd005e3d0 con 0x7fadc006d0e0 2024-02-19T19:03:22.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: mgrmap e31: smithi077.yiiqpd(active, since 91s), standbys: smithi104.sdpzmi 2024-02-19T19:03:22.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:22.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:22.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:22.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:22.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:22.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T19:03:22.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:22 smithi077 ceph-mon[29247]: pgmap v49: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s wr, 297 op/s 2024-02-19T19:03:22.277 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.256+0000 7fadcd7fa700 1 -- 172.21.15.77:0/2842263095 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fadd005e3d0 con 0x7fadc006d0e0 2024-02-19T19:03:22.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.258+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fadc006d0e0 msgr2=0x7fadc006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:22.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.258+0000 7fadd7213700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fadc006d0e0 0x7fadc006f590 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fadd00669b0 tx=0x7fadbc00b040 comp rx=0 tx=0).stop 2024-02-19T19:03:22.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.258+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 msgr2=0x7fadd00fd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:22.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.258+0000 7fadd7213700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 0x7fadd00fd570 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fadc40093b0 tx=0x7fadc4009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:22.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.259+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 shutdown_connections 2024-02-19T19:03:22.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.259+0000 7fadd7213700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fadc006d0e0 0x7fadc006f590 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.278 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.259+0000 7fadd7213700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fadd0066c70 0x7fadd00fd570 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.259+0000 7fadd7213700 1 --2- 172.21.15.77:0/2842263095 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fadd01000c0 0x7fadd00fb9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.259+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 >> 172.21.15.77:0/2842263095 conn(0x7fadd00f7520 msgr2=0x7fadd00f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:22.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.259+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 shutdown_connections 2024-02-19T19:03:22.279 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.259+0000 7fadd7213700 1 -- 172.21.15.77:0/2842263095 wait complete. 2024-02-19T19:03:22.289 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T19:03:22.393 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.374+0000 7fdf10cf3700 1 -- 172.21.15.77:0/1526990409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 msgr2=0x7fdf0c0fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:22.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.374+0000 7fdf10cf3700 1 --2- 172.21.15.77:0/1526990409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 0x7fdf0c0fd420 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fdf00004940 tx=0x7fdf00009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:22.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.374+0000 7fdf10cf3700 1 -- 172.21.15.77:0/1526990409 shutdown_connections 2024-02-19T19:03:22.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.374+0000 7fdf10cf3700 1 --2- 172.21.15.77:0/1526990409 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 0x7fdf0c0fd420 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.374+0000 7fdf10cf3700 1 --2- 172.21.15.77:0/1526990409 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf0c0fbe40 0x7fdf0c0fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.374+0000 7fdf10cf3700 1 -- 172.21.15.77:0/1526990409 >> 172.21.15.77:0/1526990409 conn(0x7fdf0c0f74e0 msgr2=0x7fdf0c0f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:22.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.374+0000 7fdf10cf3700 1 -- 172.21.15.77:0/1526990409 shutdown_connections 2024-02-19T19:03:22.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.374+0000 7fdf10cf3700 1 -- 172.21.15.77:0/1526990409 wait complete. 2024-02-19T19:03:22.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf10cf3700 1 Processor -- start 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf10cf3700 1 -- start start 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf10cf3700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf0c0fbe40 0x7fdf0c1105f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf10cf3700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 0x7fdf0c112b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf10cf3700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c066460 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf10cf3700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf0c0665a0 con 0x7fdf0c0fbe40 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf09d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 0x7fdf0c112b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf09d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 0x7fdf0c112b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46178/0 (socket says 172.21.15.77:46178) 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf09d9b700 1 -- 172.21.15.77:0/912932558 learned_addr learned my addr 172.21.15.77:0/912932558 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf0a59c700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf0c0fbe40 0x7fdf0c1105f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf09d9b700 1 -- 172.21.15.77:0/912932558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf0c0fbe40 msgr2=0x7fdf0c1105f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf09d9b700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf0c0fbe40 0x7fdf0c1105f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf09d9b700 1 -- 172.21.15.77:0/912932558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf00005040 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.375+0000 7fdf09d9b700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 0x7fdf0c112b40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdf00000c00 tx=0x7fdf00009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:22.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.376+0000 7fdefaffd700 1 -- 172.21.15.77:0/912932558 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf00021070 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.376+0000 7fdefaffd700 1 -- 172.21.15.77:0/912932558 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf0001c400 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.376+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf0c1130e0 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.376+0000 7fdefaffd700 1 -- 172.21.15.77:0/912932558 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf0000e070 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.376+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf0c113620 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.377+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf0c04bed0 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.378+0000 7fdefaffd700 1 -- 172.21.15.77:0/912932558 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7fdf0001ca50 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.378+0000 7fdefaffd700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fdef406d0e0 0x7fdef406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:22.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.379+0000 7fdefaffd700 1 -- 172.21.15.77:0/912932558 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fdf0008d6e0 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.379+0000 7fdf0a59c700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fdef406d0e0 0x7fdef406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:22.398 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.379+0000 7fdf0a59c700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fdef406d0e0 0x7fdef406f590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0660e0 tx=0x7fdefc00b040 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:22.405 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.385+0000 7fdefaffd700 1 -- 172.21.15.77:0/912932558 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fdf00057800 con 0x7fdf0c0fcfb0 2024-02-19T19:03:22.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: mgrmap e31: smithi077.yiiqpd(active, since 91s), standbys: smithi104.sdpzmi 2024-02-19T19:03:22.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:22.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.565 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:22.566 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:22 smithi104 ceph-mon[42320]: pgmap v49: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s wr, 297 op/s 2024-02-19T19:03:22.672 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.653+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf0c101990 con 0x7fdef406d0e0 2024-02-19T19:03:22.673 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.654+0000 7fdefaffd700 1 -- 172.21.15.77:0/912932558 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fdf0c101990 con 0x7fdef406d0e0 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fdef406d0e0 msgr2=0x7fdef406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fdef406d0e0 0x7fdef406f590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0660e0 tx=0x7fdefc00b040 comp rx=0 tx=0).stop 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 msgr2=0x7fdf0c112b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 0x7fdf0c112b40 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fdf00000c00 tx=0x7fdf00009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 shutdown_connections 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fdef406d0e0 0x7fdef406f590 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fdf0c0fbe40 0x7fdf0c1105f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 --2- 172.21.15.77:0/912932558 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fdf0c0fcfb0 0x7fdf0c112b40 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 >> 172.21.15.77:0/912932558 conn(0x7fdf0c0f74e0 msgr2=0x7fdf0c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 shutdown_connections 2024-02-19T19:03:22.676 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.656+0000 7fdf10cf3700 1 -- 172.21.15.77:0/912932558 wait complete. 2024-02-19T19:03:22.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.767+0000 7f250d32a700 1 -- 172.21.15.77:0/1218640357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 msgr2=0x7f25080fb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:22.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.767+0000 7f250d32a700 1 --2- 172.21.15.77:0/1218640357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 0x7f25080fb6d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f24f8004940 tx=0x7f24f8009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:22.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.767+0000 7f250d32a700 1 -- 172.21.15.77:0/1218640357 shutdown_connections 2024-02-19T19:03:22.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.767+0000 7f250d32a700 1 --2- 172.21.15.77:0/1218640357 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25080fbca0 0x7f25080fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.767+0000 7f250d32a700 1 --2- 172.21.15.77:0/1218640357 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 0x7f25080fb6d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.767+0000 7f250d32a700 1 -- 172.21.15.77:0/1218640357 >> 172.21.15.77:0/1218640357 conn(0x7f25080f6ca0 msgr2=0x7f25080f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:22.787 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.767+0000 7f250d32a700 1 -- 172.21.15.77:0/1218640357 shutdown_connections 2024-02-19T19:03:22.788 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.767+0000 7f250d32a700 1 -- 172.21.15.77:0/1218640357 wait complete. 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f250d32a700 1 Processor -- start 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f250d32a700 1 -- start start 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f250d32a700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 0x7f25080fde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f250d32a700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25080fc2a0 0x7f25080fc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f250d32a700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25080fcca0 con 0x7f25080fb2c0 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f250d32a700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25080fcde0 con 0x7f25080fc2a0 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f2506ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 0x7f25080fde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f2506ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 0x7f25080fde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46200/0 (socket says 172.21.15.77:46200) 2024-02-19T19:03:22.789 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f2506ffd700 1 -- 172.21.15.77:0/2318519282 learned_addr learned my addr 172.21.15.77:0/2318519282 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.768+0000 7f25067fc700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25080fc2a0 0x7f25080fc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f2506ffd700 1 -- 172.21.15.77:0/2318519282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25080fc2a0 msgr2=0x7f25080fc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f2506ffd700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25080fc2a0 0x7f25080fc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f2506ffd700 1 -- 172.21.15.77:0/2318519282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8005040 con 0x7f25080fb2c0 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f2506ffd700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 0x7f25080fde20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f24f8007d40 tx=0x7f24f8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f24ef7fe700 1 -- 172.21.15.77:0/2318519282 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24f8021070 con 0x7f25080fb2c0 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f24ef7fe700 1 -- 172.21.15.77:0/2318519282 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f24f801c400 con 0x7f25080fb2c0 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25080fd060 con 0x7f25080fb2c0 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f24ef7fe700 1 -- 172.21.15.77:0/2318519282 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f24f800e070 con 0x7f25080fb2c0 2024-02-19T19:03:22.790 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.769+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2508115e70 con 0x7f25080fb2c0 2024-02-19T19:03:22.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.770+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f250804bed0 con 0x7f25080fb2c0 2024-02-19T19:03:22.791 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.772+0000 7f24ef7fe700 1 -- 172.21.15.77:0/2318519282 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7f24f801c570 con 0x7f25080fb2c0 2024-02-19T19:03:22.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.772+0000 7f24ef7fe700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f24f006d0f0 0x7f24f006f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:22.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.772+0000 7f24ef7fe700 1 -- 172.21.15.77:0/2318519282 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f24f808dd30 con 0x7f25080fb2c0 2024-02-19T19:03:22.792 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.772+0000 7f25067fc700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f24f006d0f0 0x7f24f006f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:22.793 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.773+0000 7f25067fc700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f24f006d0f0 0x7f24f006f5a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009b80 tx=0x7f24fc00b040 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:22.796 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:22.776+0000 7f24ef7fe700 1 -- 172.21.15.77:0/2318519282 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f24f805ba20 con 0x7f25080fb2c0 2024-02-19T19:03:23.031 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.012+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f25080fbe20 con 0x7f24f006d0f0 2024-02-19T19:03:23.047 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.028+0000 7f24ef7fe700 1 -- 172.21.15.77:0/2318519282 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f25080fbe20 con 0x7f24f006d0f0 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (13s) 7s ago 11m 17.1M - 0.25.0 c8568f914cd2 c7b54521aa75 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (12m) 7s ago 12m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (9m) 49s ago 9m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (2m) 7s ago 10m 82.4M - 9.4.7 954c08fa6188 454ae3504841 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (4m) 7s ago 4m 14.2M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (4m) 7s ago 4m 28.0M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (4m) 49s ago 4m 14.0M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (4m) 49s ago 4m 13.0M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:8443,9283,8765 running (104s) 7s ago 13m 498M - 19.0.0-1450-gf78a58c0 f081f847070a e536fa0dfb57 2024-02-19T19:03:23.048 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283,8765 running (70s) 49s ago 9m 449M - 19.0.0-1450-gf78a58c0 f081f847070a 8af90712a2d0 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (13m) 7s ago 13m 57.3M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (9m) 49s ago 9m 42.0M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (57s) 7s ago 12m 8140k - 1.5.0 0da6a335fe13 9a8b0cdc4900 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (53s) 49s ago 9m 3409k - 1.5.0 0da6a335fe13 87a965a5c174 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (8m) 7s ago 8m 47.7M 1557M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (8m) 7s ago 8m 46.8M 1557M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (7m) 7s ago 7m 46.7M 1557M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (7m) 49s ago 7m 46.2M 2581M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (6m) 49s ago 6m 46.6M 2581M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (6m) 49s ago 6m 47.1M 2581M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T19:03:23.049 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (33s) 7s ago 9m 41.4M - 2.43.0 a07b618ecd1d 514df6ea056c 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f24f006d0f0 msgr2=0x7f24f006f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f24f006d0f0 0x7f24f006f5a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f24fc009b80 tx=0x7f24fc00b040 comp rx=0 tx=0).stop 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 msgr2=0x7f25080fde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 0x7f25080fde20 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f24f8007d40 tx=0x7f24f8009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 shutdown_connections 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f24f006d0f0 0x7f24f006f5a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f25080fb2c0 0x7f25080fde20 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 --2- 172.21.15.77:0/2318519282 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f25080fc2a0 0x7f25080fc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 >> 172.21.15.77:0/2318519282 conn(0x7f25080f6ca0 msgr2=0x7f2508101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 shutdown_connections 2024-02-19T19:03:23.051 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.031+0000 7f250d32a700 1 -- 172.21.15.77:0/2318519282 wait complete. 2024-02-19T19:03:23.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.141+0000 7faaff5d4700 1 -- 172.21.15.77:0/4133611355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fbe40 msgr2=0x7faaf80fc250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.141+0000 7faaff5d4700 1 --2- 172.21.15.77:0/4133611355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fbe40 0x7faaf80fc250 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7faaf4004940 tx=0x7faaf4009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:23.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.141+0000 7faaff5d4700 1 -- 172.21.15.77:0/4133611355 shutdown_connections 2024-02-19T19:03:23.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.141+0000 7faaff5d4700 1 --2- 172.21.15.77:0/4133611355 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faaf80fcfb0 0x7faaf80fd420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.141+0000 7faaff5d4700 1 --2- 172.21.15.77:0/4133611355 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fbe40 0x7faaf80fc250 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.161 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.141+0000 7faaff5d4700 1 -- 172.21.15.77:0/4133611355 >> 172.21.15.77:0/4133611355 conn(0x7faaf80f74e0 msgr2=0x7faaf80f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:23.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.141+0000 7faaff5d4700 1 -- 172.21.15.77:0/4133611355 shutdown_connections 2024-02-19T19:03:23.162 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.141+0000 7faaff5d4700 1 -- 172.21.15.77:0/4133611355 wait complete. 2024-02-19T19:03:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faaff5d4700 1 Processor -- start 2024-02-19T19:03:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faaff5d4700 1 -- start start 2024-02-19T19:03:23.163 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faaff5d4700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faaf80fbe40 0x7faaf810b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faaff5d4700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fcfb0 0x7faaf810bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faaff5d4700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf810c260 con 0x7faaf80fcfb0 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faaff5d4700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf810cb00 con 0x7faaf80fbe40 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faafcb6f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fcfb0 0x7faaf810bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faafcb6f700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fcfb0 0x7faaf810bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46216/0 (socket says 172.21.15.77:46216) 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.142+0000 7faafcb6f700 1 -- 172.21.15.77:0/1261944073 learned_addr learned my addr 172.21.15.77:0/1261944073 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faafd370700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faaf80fbe40 0x7faaf810b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faafcb6f700 1 -- 172.21.15.77:0/1261944073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faaf80fbe40 msgr2=0x7faaf810b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faafcb6f700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faaf80fbe40 0x7faaf810b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faafcb6f700 1 -- 172.21.15.77:0/1261944073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf4005040 con 0x7faaf80fcfb0 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faafcb6f700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fcfb0 0x7faaf810bc60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faae800c8c0 tx=0x7faae80044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faaedffb700 1 -- 172.21.15.77:0/1261944073 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faae800a760 con 0x7faaf80fcfb0 2024-02-19T19:03:23.164 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faaedffb700 1 -- 172.21.15.77:0/1261944073 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faae800ad90 con 0x7faaf80fcfb0 2024-02-19T19:03:23.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf810cde0 con 0x7faaf80fcfb0 2024-02-19T19:03:23.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.143+0000 7faaedffb700 1 -- 172.21.15.77:0/1261944073 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faae80113a0 con 0x7faaf80fcfb0 2024-02-19T19:03:23.165 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.144+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf810d320 con 0x7faaf80fcfb0 2024-02-19T19:03:23.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.145+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf804bed0 con 0x7faaf80fcfb0 2024-02-19T19:03:23.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.145+0000 7faaedffb700 1 -- 172.21.15.77:0/1261944073 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7faae8012750 con 0x7faaf80fcfb0 2024-02-19T19:03:23.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.146+0000 7faaedffb700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7faae406f2e0 0x7faae4071790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:23.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.146+0000 7faaedffb700 1 -- 172.21.15.77:0/1261944073 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7faae808c490 con 0x7faaf80fcfb0 2024-02-19T19:03:23.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.146+0000 7faafd370700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7faae406f2e0 0x7faae4071790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:23.166 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.146+0000 7faafd370700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7faae406f2e0 0x7faae4071790 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faaf4004f90 tx=0x7faaf4009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:23.170 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.151+0000 7faaedffb700 1 -- 172.21.15.77:0/1261944073 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7faae805a180 con 0x7faaf80fcfb0 2024-02-19T19:03:23.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.459+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7faaf8103830 con 0x7faaf80fcfb0 2024-02-19T19:03:23.479 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.460+0000 7faaedffb700 1 -- 172.21.15.77:0/1261944073 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7faae8059d10 con 0x7faaf80fcfb0 2024-02-19T19:03:23.479 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:03:23.479 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T19:03:23.479 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:03:23.479 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 12, 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T19:03:23.480 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:03:23.481 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7faae406f2e0 msgr2=0x7faae4071790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7faae406f2e0 0x7faae4071790 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faaf4004f90 tx=0x7faaf4009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fcfb0 msgr2=0x7faaf810bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fcfb0 0x7faaf810bc60 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7faae800c8c0 tx=0x7faae80044c0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 shutdown_connections 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7faae406f2e0 0x7faae4071790 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7faaf80fbe40 0x7faaf810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 --2- 172.21.15.77:0/1261944073 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7faaf80fcfb0 0x7faaf810bc60 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 >> 172.21.15.77:0/1261944073 conn(0x7faaf80f74e0 msgr2=0x7faaf8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 shutdown_connections 2024-02-19T19:03:23.482 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.462+0000 7faaff5d4700 1 -- 172.21.15.77:0/1261944073 wait complete. 2024-02-19T19:03:23.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.574+0000 7fd7bac34700 1 -- 172.21.15.77:0/2808902773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 msgr2=0x7fd7b4102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.574+0000 7fd7bac34700 1 --2- 172.21.15.77:0/2808902773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 0x7fd7b4102520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd7a0004940 tx=0x7fd7a0009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:23.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.574+0000 7fd7bac34700 1 -- 172.21.15.77:0/2808902773 shutdown_connections 2024-02-19T19:03:23.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.574+0000 7fd7bac34700 1 --2- 172.21.15.77:0/2808902773 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 0x7fd7b4102520 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.574+0000 7fd7bac34700 1 --2- 172.21.15.77:0/2808902773 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7b4066380 0x7fd7b40ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.574+0000 7fd7bac34700 1 -- 172.21.15.77:0/2808902773 >> 172.21.15.77:0/2808902773 conn(0x7fd7b40f74e0 msgr2=0x7fd7b40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:23.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.574+0000 7fd7bac34700 1 -- 172.21.15.77:0/2808902773 shutdown_connections 2024-02-19T19:03:23.594 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.574+0000 7fd7bac34700 1 -- 172.21.15.77:0/2808902773 wait complete. 2024-02-19T19:03:23.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.575+0000 7fd7bac34700 1 Processor -- start 2024-02-19T19:03:23.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.575+0000 7fd7bac34700 1 -- start start 2024-02-19T19:03:23.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7bac34700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7b4066380 0x7fd7b410d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:23.596 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7bac34700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 0x7fd7b410deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7bac34700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b410e4b0 con 0x7fd7b4100120 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7bac34700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7b410ed50 con 0x7fd7b4066380 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 0x7fd7b410deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b3fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 0x7fd7b410deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:46222/0 (socket says 172.21.15.77:46222) 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b3fff700 1 -- 172.21.15.77:0/1064139869 learned_addr learned my addr 172.21.15.77:0/1064139869 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b89d0700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7b4066380 0x7fd7b410d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b3fff700 1 -- 172.21.15.77:0/1064139869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7b4066380 msgr2=0x7fd7b410d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b3fff700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7b4066380 0x7fd7b410d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b3fff700 1 -- 172.21.15.77:0/1064139869 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a0005040 con 0x7fd7b4100120 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b3fff700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 0x7fd7b410deb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000b3b0 tx=0x7fd7a0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.576+0000 7fd7b17fa700 1 -- 172.21.15.77:0/1064139869 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7a000e040 con 0x7fd7b4100120 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.577+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b410efd0 con 0x7fd7b4100120 2024-02-19T19:03:23.597 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.577+0000 7fd7b17fa700 1 -- 172.21.15.77:0/1064139869 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd7a000bcc0 con 0x7fd7b4100120 2024-02-19T19:03:23.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.577+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b410f510 con 0x7fd7b4100120 2024-02-19T19:03:23.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.577+0000 7fd7b17fa700 1 -- 172.21.15.77:0/1064139869 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7a001c630 con 0x7fd7b4100120 2024-02-19T19:03:23.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.578+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b404bed0 con 0x7fd7b4100120 2024-02-19T19:03:23.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.578+0000 7fd7b17fa700 1 -- 172.21.15.77:0/1064139869 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7fd7a00139f0 con 0x7fd7b4100120 2024-02-19T19:03:23.598 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.579+0000 7fd7b17fa700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7a406d150 0x7fd7a406f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:23.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.579+0000 7fd7b17fa700 1 -- 172.21.15.77:0/1064139869 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd7a008d980 con 0x7fd7b4100120 2024-02-19T19:03:23.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.579+0000 7fd7b89d0700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7a406d150 0x7fd7a406f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:23.599 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.579+0000 7fd7b89d0700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7a406d150 0x7fd7a406f600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8007fd0 tx=0x7fd7a8009040 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:23.603 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.583+0000 7fd7b17fa700 1 -- 172.21.15.77:0/1064139869 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd7a005b5d0 con 0x7fd7b4100120 2024-02-19T19:03:23.912 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.893+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7b410f900 con 0x7fd7a406d150 2024-02-19T19:03:23.913 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.894+0000 7fd7b17fa700 1 -- 172.21.15.77:0/1064139869 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fd7b410f900 con 0x7fd7a406d150 2024-02-19T19:03:23.913 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:03:23.913 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading grafana daemons", 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T19:03:23.914 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:03:23.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7a406d150 msgr2=0x7fd7a406f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7a406d150 0x7fd7a406f600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd7a8007fd0 tx=0x7fd7a8009040 comp rx=0 tx=0).stop 2024-02-19T19:03:23.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 msgr2=0x7fd7b410deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:23.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 0x7fd7b410deb0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000b3b0 tx=0x7fd7a0009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:23.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 shutdown_connections 2024-02-19T19:03:23.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd7a406d150 0x7fd7a406f600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7b4066380 0x7fd7b410d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.916 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 --2- 172.21.15.77:0/1064139869 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7b4100120 0x7fd7b410deb0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:23.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 >> 172.21.15.77:0/1064139869 conn(0x7fd7b40f74e0 msgr2=0x7fd7b40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:23.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 shutdown_connections 2024-02-19T19:03:23.917 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:23.896+0000 7fd7bac34700 1 -- 172.21.15.77:0/1064139869 wait complete. 2024-02-19T19:03:24.239 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:23 smithi077 ceph-mon[29247]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:24.240 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:23 smithi077 ceph-mon[29247]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:24.240 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:23 smithi077 ceph-mon[29247]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:24.240 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:23 smithi077 ceph-mon[29247]: from='client.? 172.21.15.77:0/1261944073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:24.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:23 smithi104 ceph-mon[42320]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:24.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:23 smithi104 ceph-mon[42320]: from='client.14812 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:24.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:23 smithi104 ceph-mon[42320]: from='client.14816 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:24.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:23 smithi104 ceph-mon[42320]: from='client.? 172.21.15.77:0/1261944073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-02-19T19:03:25.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:24 smithi077 ceph-mon[29247]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:25.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:24 smithi077 ceph-mon[29247]: pgmap v50: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s wr, 297 op/s 2024-02-19T19:03:25.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:24 smithi104 ceph-mon[42320]: from='client.14824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-19T19:03:25.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:24 smithi104 ceph-mon[42320]: pgmap v50: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s wr, 297 op/s 2024-02-19T19:03:26.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:26 smithi077 ceph-mon[29247]: pgmap v51: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s wr, 297 op/s 2024-02-19T19:03:26.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:26 smithi104 ceph-mon[42320]: pgmap v51: 65 pgs: 65 active+clean; 57 MiB data, 1.9 GiB used, 535 GiB / 536 GiB avail; 4.9 MiB/s wr, 297 op/s 2024-02-19T19:03:29.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:28 smithi077 ceph-mon[29247]: pgmap v52: 65 pgs: 65 active+clean; 69 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s wr, 25 op/s 2024-02-19T19:03:29.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:28 smithi104 ceph-mon[42320]: pgmap v52: 65 pgs: 65 active+clean; 69 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s wr, 25 op/s 2024-02-19T19:03:30.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:30 smithi077 ceph-mon[29247]: pgmap v53: 65 pgs: 65 active+clean; 69 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s wr, 25 op/s 2024-02-19T19:03:30.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:30 smithi104 ceph-mon[42320]: pgmap v53: 65 pgs: 65 active+clean; 69 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s wr, 25 op/s 2024-02-19T19:03:32.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:32 smithi077 ceph-mon[29247]: pgmap v54: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 6.9 MiB/s wr, 645 op/s 2024-02-19T19:03:32.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:32 smithi104 ceph-mon[42320]: pgmap v54: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 6.9 MiB/s wr, 645 op/s 2024-02-19T19:03:34.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:33 smithi077 ceph-mon[29247]: Upgrade: Updating grafana.smithi077 2024-02-19T19:03:34.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:33 smithi077 ceph-mon[29247]: Deploying daemon grafana.smithi077 on smithi077 2024-02-19T19:03:34.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:33 smithi104 ceph-mon[42320]: Upgrade: Updating grafana.smithi077 2024-02-19T19:03:34.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:33 smithi104 ceph-mon[42320]: Deploying daemon grafana.smithi077 on smithi077 2024-02-19T19:03:35.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:34 smithi077 ceph-mon[29247]: pgmap v55: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.5 MiB/s wr, 636 op/s 2024-02-19T19:03:35.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:34 smithi104 ceph-mon[42320]: pgmap v55: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.5 MiB/s wr, 636 op/s 2024-02-19T19:03:36.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:35 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:36.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:35 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-19T19:03:37.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:36 smithi077 ceph-mon[29247]: pgmap v56: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.5 MiB/s wr, 636 op/s 2024-02-19T19:03:37.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:36 smithi104 ceph-mon[42320]: pgmap v56: 65 pgs: 65 active+clean; 118 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 5.5 MiB/s wr, 636 op/s 2024-02-19T19:03:39.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:38 smithi077 ceph-mon[29247]: pgmap v57: 65 pgs: 65 active+clean; 144 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.8 MiB/s wr, 952 op/s 2024-02-19T19:03:39.314 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:38 smithi104 ceph-mon[42320]: pgmap v57: 65 pgs: 65 active+clean; 144 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 7.8 MiB/s wr, 952 op/s 2024-02-19T19:03:40.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:40 smithi077 ceph-mon[29247]: pgmap v58: 65 pgs: 65 active+clean; 144 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.8 MiB/s wr, 935 op/s 2024-02-19T19:03:40.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:40 smithi104 ceph-mon[42320]: pgmap v58: 65 pgs: 65 active+clean; 144 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 6.8 MiB/s wr, 935 op/s 2024-02-19T19:03:42.310 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:42 smithi077 ceph-mon[29247]: pgmap v59: 65 pgs: 65 active+clean; 158 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 8.0 MiB/s wr, 950 op/s 2024-02-19T19:03:42.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:42 smithi104 ceph-mon[42320]: pgmap v59: 65 pgs: 65 active+clean; 158 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 8.0 MiB/s wr, 950 op/s 2024-02-19T19:03:44.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:44 smithi077 ceph-mon[29247]: pgmap v60: 65 pgs: 65 active+clean; 158 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.5 MiB/s wr, 330 op/s 2024-02-19T19:03:44.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:44 smithi104 ceph-mon[42320]: pgmap v60: 65 pgs: 65 active+clean; 158 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.5 MiB/s wr, 330 op/s 2024-02-19T19:03:45.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:45 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:45.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:45 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:45.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:45 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:45.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:45 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:45.561 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:45 smithi077 ceph-mon[29247]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:45 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:45 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:45 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-19T19:03:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:45 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:45.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:45 smithi104 ceph-mon[42320]: from='mgr.14714 172.21.15.77:0/3348857227' entity='mgr.smithi077.yiiqpd' 2024-02-19T19:03:46.367 INFO:tasks.workunit.client.1.smithi104.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-19T19:03:46.367 INFO:tasks.workunit.client.1.smithi104.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-02-19T19:03:46.367 INFO:tasks.workunit.client.1.smithi104.stderr:+ make 2024-02-19T19:03:46.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:03:46 smithi077 ceph-mon[29247]: pgmap v61: 65 pgs: 65 active+clean; 158 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.5 MiB/s wr, 330 op/s 2024-02-19T19:03:46.564 INFO:journalctl@ceph.mon.smithi104.smithi104.stdout:Feb 19 19:03:46 smithi104 ceph-mon[42320]: pgmap v61: 65 pgs: 65 active+clean; 158 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 3.5 MiB/s wr, 330 op/s 2024-02-19T19:03:46.857 INFO:tasks.workunit.client.1.smithi104.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-02-19T19:03:47.008 INFO:tasks.workunit.client.0.smithi077.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-02-19T19:03:47.008 INFO:tasks.workunit.client.0.smithi077.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-02-19T19:03:47.008 INFO:tasks.workunit.client.0.smithi077.stderr:+ make 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.014+0000 7fa9c8653700 1 -- 172.21.15.77:0/3156727914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 msgr2=0x7fa9c0102780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.014+0000 7fa9c8653700 1 --2- 172.21.15.77:0/3156727914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 0x7fa9c0102780 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0004940 tx=0x7fa9b0009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.014+0000 7fa9c8653700 1 -- 172.21.15.77:0/3156727914 shutdown_connections 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.014+0000 7fa9c8653700 1 --2- 172.21.15.77:0/3156727914 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 0x7fa9c0102780 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.014+0000 7fa9c8653700 1 --2- 172.21.15.77:0/3156727914 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00fda80 0x7fa9c00ffe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.014+0000 7fa9c8653700 1 -- 172.21.15.77:0/3156727914 >> 172.21.15.77:0/3156727914 conn(0x7fa9c00f7520 msgr2=0x7fa9c00f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.014+0000 7fa9c8653700 1 -- 172.21.15.77:0/3156727914 shutdown_connections 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.014+0000 7fa9c8653700 1 -- 172.21.15.77:0/3156727914 wait complete. 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c8653700 1 Processor -- start 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c8653700 1 -- start start 2024-02-19T19:03:54.036 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c8653700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00fda80 0x7fa9c010d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c8653700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 0x7fa9c010df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c8653700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c010e5c0 con 0x7fa9c01003a0 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c8653700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c010ee00 con 0x7fa9c00fda80 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c5bee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 0x7fa9c010df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c5bee700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 0x7fa9c010df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49414/0 (socket says 172.21.15.77:49414) 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.015+0000 7fa9c5bee700 1 -- 172.21.15.77:0/4260547258 learned_addr learned my addr 172.21.15.77:0/4260547258 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9c5bee700 1 -- 172.21.15.77:0/4260547258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00fda80 msgr2=0x7fa9c010d9f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9c5bee700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00fda80 0x7fa9c010d9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9c5bee700 1 -- 172.21.15.77:0/4260547258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b0005040 con 0x7fa9c01003a0 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9c5bee700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 0x7fa9c010df30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0004890 tx=0x7fa9b0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/4260547258 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9b000e030 con 0x7fa9c01003a0 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/4260547258 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa9b0004290 con 0x7fa9c01003a0 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c010f080 con 0x7fa9c01003a0 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/4260547258 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9b001c660 con 0x7fa9c01003a0 2024-02-19T19:03:54.037 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.016+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c010f5c0 con 0x7fa9c01003a0 2024-02-19T19:03:54.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.017+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c004bed0 con 0x7fa9c01003a0 2024-02-19T19:03:54.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.017+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/4260547258 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7fa9b000bbf0 con 0x7fa9c01003a0 2024-02-19T19:03:54.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.018+0000 7fa9b6ffd700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fa9ac06d0e0 0x7fa9ac06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.018+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/4260547258 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fa9b008deb0 con 0x7fa9c01003a0 2024-02-19T19:03:54.038 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.018+0000 7fa9c63ef700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fa9ac06d0e0 0x7fa9ac06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:54.039 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.018+0000 7fa9c63ef700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fa9ac06d0e0 0x7fa9ac06f590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009740 tx=0x7fa9bc006c60 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:54.042 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.023+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/4260547258 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fa9b005bb00 con 0x7fa9c01003a0 2024-02-19T19:03:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.231+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9c010fad0 con 0x7fa9ac06d0e0 2024-02-19T19:03:54.251 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.232+0000 7fa9b6ffd700 1 -- 172.21.15.77:0/4260547258 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fa9c010fad0 con 0x7fa9ac06d0e0 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fa9ac06d0e0 msgr2=0x7fa9ac06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fa9ac06d0e0 0x7fa9ac06f590 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc009740 tx=0x7fa9bc006c60 comp rx=0 tx=0).stop 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 msgr2=0x7fa9c010df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 0x7fa9c010df30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa9b0004890 tx=0x7fa9b0009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 shutdown_connections 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fa9ac06d0e0 0x7fa9ac06f590 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fa9c00fda80 0x7fa9c010d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 --2- 172.21.15.77:0/4260547258 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fa9c01003a0 0x7fa9c010df30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 >> 172.21.15.77:0/4260547258 conn(0x7fa9c00f7520 msgr2=0x7fa9c00f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 shutdown_connections 2024-02-19T19:03:54.254 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.234+0000 7fa9c8653700 1 -- 172.21.15.77:0/4260547258 wait complete. 2024-02-19T19:03:54.264 INFO:teuthology.orchestra.run.smithi077.stdout:true 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/1624728065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 msgr2=0x7fd7a40fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 --2- 172.21.15.77:0/1624728065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 0x7fd7a40fd420 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd798004940 tx=0x7fd798009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/1624728065 shutdown_connections 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 --2- 172.21.15.77:0/1624728065 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 0x7fd7a40fd420 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 --2- 172.21.15.77:0/1624728065 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7a40fbe40 0x7fd7a40fc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/1624728065 >> 172.21.15.77:0/1624728065 conn(0x7fd7a40f74e0 msgr2=0x7fd7a40f9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/1624728065 shutdown_connections 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/1624728065 wait complete. 2024-02-19T19:03:54.359 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 Processor -- start 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.338+0000 7fd7ab8e5700 1 -- start start 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7ab8e5700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7a40fbe40 0x7fd7a4112b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7ab8e5700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 0x7fd7a4115060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7ab8e5700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a4066350 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7ab8e5700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7a4066490 con 0x7fd7a40fbe40 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a8e80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 0x7fd7a4115060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a8e80700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 0x7fd7a4115060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49438/0 (socket says 172.21.15.77:49438) 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a8e80700 1 -- 172.21.15.77:0/2600105990 learned_addr learned my addr 172.21.15.77:0/2600105990 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a8e80700 1 -- 172.21.15.77:0/2600105990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7a40fbe40 msgr2=0x7fd7a4112b10 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a8e80700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7a40fbe40 0x7fd7a4112b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.360 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a8e80700 1 -- 172.21.15.77:0/2600105990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd798005040 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a8e80700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 0x7fd7a4115060 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd798004890 tx=0x7fd798009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:54.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a1ffb700 1 -- 172.21.15.77:0/2600105990 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd79800e030 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a1ffb700 1 -- 172.21.15.77:0/2600105990 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd798004290 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a41155a0 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.361 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7a1ffb700 1 -- 172.21.15.77:0/2600105990 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd79801c660 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.339+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a4115a00 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.340+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a404bed0 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.341+0000 7fd7a1ffb700 1 -- 172.21.15.77:0/2600105990 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7fd79800bbf0 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.341+0000 7fd7a1ffb700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd78c06d0e0 0x7fd78c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.341+0000 7fd7a1ffb700 1 -- 172.21.15.77:0/2600105990 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7fd79808dfb0 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.341+0000 7fd7a9681700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd78c06d0e0 0x7fd78c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:54.362 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.342+0000 7fd7a9681700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd78c06d0e0 0x7fd78c06f590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40660e0 tx=0x7fd79c009370 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:54.365 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.346+0000 7fd7a1ffb700 1 -- 172.21.15.77:0/2600105990 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7fd798058050 con 0x7fd7a40fcfb0 2024-02-19T19:03:54.578 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.558+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7a4101990 con 0x7fd78c06d0e0 2024-02-19T19:03:54.579 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.560+0000 7fd7a1ffb700 1 -- 172.21.15.77:0/2600105990 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7fd7a4101990 con 0x7fd78c06d0e0 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.561+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd78c06d0e0 msgr2=0x7fd78c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.561+0000 7fd7ab8e5700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd78c06d0e0 0x7fd78c06f590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd7a40660e0 tx=0x7fd79c009370 comp rx=0 tx=0).stop 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.561+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 msgr2=0x7fd7a4115060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.561+0000 7fd7ab8e5700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 0x7fd7a4115060 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd798004890 tx=0x7fd798009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.562+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 shutdown_connections 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.562+0000 7fd7ab8e5700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7fd78c06d0e0 0x7fd78c06f590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.562+0000 7fd7ab8e5700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7fd7a40fbe40 0x7fd7a4112b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.562+0000 7fd7ab8e5700 1 --2- 172.21.15.77:0/2600105990 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7fd7a40fcfb0 0x7fd7a4115060 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.562+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 >> 172.21.15.77:0/2600105990 conn(0x7fd7a40f74e0 msgr2=0x7fd7a4100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.562+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 shutdown_connections 2024-02-19T19:03:54.582 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.562+0000 7fd7ab8e5700 1 -- 172.21.15.77:0/2600105990 wait complete. 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.662+0000 7f96b2788700 1 -- 172.21.15.77:0/1072623582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fe0e0 msgr2=0x7f96ac0fe550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.662+0000 7f96b2788700 1 --2- 172.21.15.77:0/1072623582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fe0e0 0x7f96ac0fe550 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f96a0004940 tx=0x7f96a0009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.662+0000 7f96b2788700 1 -- 172.21.15.77:0/1072623582 shutdown_connections 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.662+0000 7f96b2788700 1 --2- 172.21.15.77:0/1072623582 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fe0e0 0x7f96ac0fe550 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.662+0000 7f96b2788700 1 --2- 172.21.15.77:0/1072623582 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac0fcef0 0x7f96ac0fd320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.662+0000 7f96b2788700 1 -- 172.21.15.77:0/1072623582 >> 172.21.15.77:0/1072623582 conn(0x7f96ac0f8620 msgr2=0x7f96ac0faa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.662+0000 7f96b2788700 1 -- 172.21.15.77:0/1072623582 shutdown_connections 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.662+0000 7f96b2788700 1 -- 172.21.15.77:0/1072623582 wait complete. 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96b2788700 1 Processor -- start 2024-02-19T19:03:54.683 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96b2788700 1 -- start start 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96b2788700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fcef0 0x7f96ac10d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96b2788700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac0fe0e0 0x7f96ac10df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96b2788700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac10e5c0 con 0x7f96ac0fcef0 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96b2788700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96ac10ee00 con 0x7f96ac0fe0e0 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fcef0 0x7f96ac10d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96abfff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fcef0 0x7f96ac10d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49456/0 (socket says 172.21.15.77:49456) 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.663+0000 7f96abfff700 1 -- 172.21.15.77:0/1631851352 learned_addr learned my addr 172.21.15.77:0/1631851352 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96abfff700 1 -- 172.21.15.77:0/1631851352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac0fe0e0 msgr2=0x7f96ac10df30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T19:03:54.685 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96abfff700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac0fe0e0 0x7f96ac10df30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96abfff700 1 -- 172.21.15.77:0/1631851352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a0005040 con 0x7f96ac0fcef0 2024-02-19T19:03:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96abfff700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fcef0 0x7f96ac10d9f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f969c00b760 tx=0x7f969c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96a8ff9700 1 -- 172.21.15.77:0/1631851352 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f969c00f6f0 con 0x7f96ac0fcef0 2024-02-19T19:03:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96a8ff9700 1 -- 172.21.15.77:0/1631851352 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f969c00fd20 con 0x7f96ac0fcef0 2024-02-19T19:03:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ac10f0e0 con 0x7f96ac0fcef0 2024-02-19T19:03:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96a8ff9700 1 -- 172.21.15.77:0/1631851352 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f969c011e40 con 0x7f96ac0fcef0 2024-02-19T19:03:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.664+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ac10f620 con 0x7f96ac0fcef0 2024-02-19T19:03:54.686 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.665+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96ac04bed0 con 0x7f96ac0fcef0 2024-02-19T19:03:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.666+0000 7f96a8ff9700 1 -- 172.21.15.77:0/1631851352 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7f969c011670 con 0x7f96ac0fcef0 2024-02-19T19:03:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.666+0000 7f96a8ff9700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f969406d0e0 0x7f969406f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.667+0000 7f96a8ff9700 1 -- 172.21.15.77:0/1631851352 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f969c08c220 con 0x7f96ac0fcef0 2024-02-19T19:03:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.667+0000 7f96ab7fe700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f969406d0e0 0x7f969406f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:54.687 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.667+0000 7f96ab7fe700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f969406d0e0 0x7f969406f590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96a0004890 tx=0x7f96a0009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:54.691 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.671+0000 7f96a8ff9700 1 -- 172.21.15.77:0/1631851352 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f969c059e70 con 0x7f96ac0fcef0 2024-02-19T19:03:54.890 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.871+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f96ac102ac0 con 0x7f969406d0e0 2024-02-19T19:03:54.900 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.880+0000 7f96a8ff9700 1 -- 172.21.15.77:0/1631851352 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3520 (secure 0 0 0) 0x7f96ac102ac0 con 0x7f969406d0e0 2024-02-19T19:03:54.900 INFO:teuthology.orchestra.run.smithi077.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-19T19:03:54.900 INFO:teuthology.orchestra.run.smithi077.stdout:alertmanager.smithi077 smithi077 *:9093,9094 running (45s) 2s ago 12m 17.1M - 0.25.0 c8568f914cd2 c7b54521aa75 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi077 smithi077 running (12m) 2s ago 12m 6660k - 17.2.7-521-gc80e7168 10b0c032c913 908a3e2da7f8 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:crash.smithi104 smithi104 running (10m) 81s ago 10m 6665k - 17.2.7-521-gc80e7168 10b0c032c913 2dfbf58c6ebf 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:grafana.smithi077 smithi077 *:3000 running (11s) 2s ago 11m 56.3M - 9.4.12 a0ea5e3c46d9 aabda23ef183 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.jmjalp smithi077 running (5m) 2s ago 5m 14.4M - 17.2.7-521-gc80e7168 10b0c032c913 b159ce0cdc34 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi077.ovassg smithi077 running (5m) 2s ago 5m 192M - 17.2.7-521-gc80e7168 10b0c032c913 d7fea4fb72a3 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.rudtoq smithi104 running (5m) 81s ago 5m 14.0M - 17.2.7-521-gc80e7168 10b0c032c913 120dc0e39496 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:mds.cephfs.smithi104.wpcqst smithi104 running (5m) 81s ago 5m 13.0M - 17.2.7-521-gc80e7168 10b0c032c913 4dcb7aa5033b 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi077.yiiqpd smithi077 *:8443,9283,8765 running (2m) 2s ago 13m 500M - 19.0.0-1450-gf78a58c0 f081f847070a e536fa0dfb57 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:mgr.smithi104.sdpzmi smithi104 *:8443,9283,8765 running (102s) 81s ago 10m 449M - 19.0.0-1450-gf78a58c0 f081f847070a 8af90712a2d0 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi077 smithi077 running (14m) 2s ago 14m 58.5M 2048M 17.2.7-521-gc80e7168 10b0c032c913 7b41d21d0c00 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:mon.smithi104 smithi104 running (9m) 81s ago 9m 42.0M 2048M 17.2.7-521-gc80e7168 10b0c032c913 0fd04033894f 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi077 smithi077 *:9100 running (89s) 2s ago 12m 7922k - 1.5.0 0da6a335fe13 9a8b0cdc4900 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:node-exporter.smithi104 smithi104 *:9100 running (84s) 81s ago 10m 3409k - 1.5.0 0da6a335fe13 87a965a5c174 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:osd.0 smithi077 running (9m) 2s ago 9m 118M 1557M 17.2.7-521-gc80e7168 10b0c032c913 7a663ccf6c07 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:osd.1 smithi077 running (8m) 2s ago 8m 127M 1557M 17.2.7-521-gc80e7168 10b0c032c913 c0fec34c6b1c 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:osd.2 smithi077 running (8m) 2s ago 8m 106M 1557M 17.2.7-521-gc80e7168 10b0c032c913 0f448003509d 2024-02-19T19:03:54.901 INFO:teuthology.orchestra.run.smithi077.stdout:osd.3 smithi104 running (7m) 81s ago 7m 46.2M 2581M 17.2.7-521-gc80e7168 10b0c032c913 581041091e6c 2024-02-19T19:03:54.902 INFO:teuthology.orchestra.run.smithi077.stdout:osd.4 smithi104 running (7m) 81s ago 7m 46.6M 2581M 17.2.7-521-gc80e7168 10b0c032c913 bf1d8f6dd6e7 2024-02-19T19:03:54.902 INFO:teuthology.orchestra.run.smithi077.stdout:osd.5 smithi104 running (6m) 81s ago 6m 47.1M 2581M 17.2.7-521-gc80e7168 10b0c032c913 fb07c4155175 2024-02-19T19:03:54.902 INFO:teuthology.orchestra.run.smithi077.stdout:prometheus.smithi077 smithi077 *:9095 running (65s) 2s ago 10m 44.0M - 2.43.0 a07b618ecd1d 514df6ea056c 2024-02-19T19:03:54.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.882+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f969406d0e0 msgr2=0x7f969406f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.882+0000 7f96b2788700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f969406d0e0 0x7f969406f590 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f96a0004890 tx=0x7f96a0009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:54.902 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.882+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fcef0 msgr2=0x7f96ac10d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.882+0000 7f96b2788700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fcef0 0x7f96ac10d9f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f969c00b760 tx=0x7f969c0091a0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.883+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 shutdown_connections 2024-02-19T19:03:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.883+0000 7f96b2788700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f969406d0e0 0x7f969406f590 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.883+0000 7f96b2788700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f96ac0fcef0 0x7f96ac10d9f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.883+0000 7f96b2788700 1 --2- 172.21.15.77:0/1631851352 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f96ac0fe0e0 0x7f96ac10df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.883+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 >> 172.21.15.77:0/1631851352 conn(0x7f96ac0f8620 msgr2=0x7f96ac1013a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.883+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 shutdown_connections 2024-02-19T19:03:54.903 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.883+0000 7f96b2788700 1 -- 172.21.15.77:0/1631851352 wait complete. 2024-02-19T19:03:55.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.981+0000 7f685cecb700 1 -- 172.21.15.77:0/3506919039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fc0a0 msgr2=0x7f68580fc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:55.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.981+0000 7f685cecb700 1 --2- 172.21.15.77:0/3506919039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fc0a0 0x7f68580fc510 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f684c004940 tx=0x7f684c009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:55.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 -- 172.21.15.77:0/3506919039 shutdown_connections 2024-02-19T19:03:55.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 --2- 172.21.15.77:0/3506919039 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fc0a0 0x7f68580fc510 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 --2- 172.21.15.77:0/3506919039 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68580fd430 0x7f68580fd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.002 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 -- 172.21.15.77:0/3506919039 >> 172.21.15.77:0/3506919039 conn(0x7f68580f6ca0 msgr2=0x7f68580f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 -- 172.21.15.77:0/3506919039 shutdown_connections 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 -- 172.21.15.77:0/3506919039 wait complete. 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 Processor -- start 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 -- start start 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68580fc0a0 0x7f68581107b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fd430 0x7f6858112d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580fbe20 con 0x7f68580fd430 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f685cecb700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580fbf60 con 0x7f68580fc0a0 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f6855d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fd430 0x7f6858112d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f6855d9b700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fd430 0x7f6858112d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49466/0 (socket says 172.21.15.77:49466) 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.982+0000 7f6855d9b700 1 -- 172.21.15.77:0/3907213526 learned_addr learned my addr 172.21.15.77:0/3907213526 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.983+0000 7f6855d9b700 1 -- 172.21.15.77:0/3907213526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68580fc0a0 msgr2=0x7f68581107b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.983+0000 7f6855d9b700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68580fc0a0 0x7f68581107b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.003 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.983+0000 7f6855d9b700 1 -- 172.21.15.77:0/3907213526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c005040 con 0x7f68580fd430 2024-02-19T19:03:55.004 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.983+0000 7f6855d9b700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fd430 0x7f6858112d00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f684c004070 tx=0x7f684c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.983+0000 7f6846ffd700 1 -- 172.21.15.77:0/3907213526 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f684c00e030 con 0x7f68580fd430 2024-02-19T19:03:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.983+0000 7f6846ffd700 1 -- 172.21.15.77:0/3907213526 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f684c009430 con 0x7f68580fd430 2024-02-19T19:03:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.983+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6858113380 con 0x7f68580fd430 2024-02-19T19:03:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.983+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68581138c0 con 0x7f68580fd430 2024-02-19T19:03:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.984+0000 7f6846ffd700 1 -- 172.21.15.77:0/3907213526 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f684c01c400 con 0x7f68580fd430 2024-02-19T19:03:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.984+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685804bed0 con 0x7f68580fd430 2024-02-19T19:03:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.985+0000 7f6846ffd700 1 -- 172.21.15.77:0/3907213526 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7f684c00bbf0 con 0x7f68580fd430 2024-02-19T19:03:55.005 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.985+0000 7f6846ffd700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f684006d0e0 0x7f684006f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.985+0000 7f6846ffd700 1 -- 172.21.15.77:0/3907213526 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f684c08da70 con 0x7f68580fd430 2024-02-19T19:03:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.985+0000 7f685659c700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f684006d0e0 0x7f684006f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:55.006 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.986+0000 7f685659c700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f684006d0e0 0x7f684006f590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f68580fba10 tx=0x7f6848006c60 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:55.009 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:54.990+0000 7f6846ffd700 1 -- 172.21.15.77:0/3907213526 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f684c057c30 con 0x7f68580fd430 2024-02-19T19:03:55.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.267+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f6858113da0 con 0x7f68580fd430 2024-02-19T19:03:55.287 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.268+0000 7f6846ffd700 1 -- 172.21.15.77:0/3907213526 <== mon.0 v2:172.21.15.77:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+739 (secure 0 0 0) 0x7f684c05b2f0 con 0x7f68580fd430 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: "mon": { 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 2 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr": { 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: "osd": { 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 6 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:03:55.288 INFO:teuthology.orchestra.run.smithi077.stdout: "mds": { 2024-02-19T19:03:55.289 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 4 2024-02-19T19:03:55.289 INFO:teuthology.orchestra.run.smithi077.stdout: }, 2024-02-19T19:03:55.289 INFO:teuthology.orchestra.run.smithi077.stdout: "overall": { 2024-02-19T19:03:55.289 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 17.2.7-521-gc80e7168 (c80e7168938c48b49e5ce2e002f48bf150b5f4a4) quincy (stable)": 12, 2024-02-19T19:03:55.289 INFO:teuthology.orchestra.run.smithi077.stdout: "ceph version 19.0.0-1450-gf78a58c0 (f78a58c0ffd401d1493058a1022c35f011d65275) squid (dev)": 2 2024-02-19T19:03:55.289 INFO:teuthology.orchestra.run.smithi077.stdout: } 2024-02-19T19:03:55.289 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f684006d0e0 msgr2=0x7f684006f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f684006d0e0 0x7f684006f590 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f68580fba10 tx=0x7f6848006c60 comp rx=0 tx=0).stop 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fd430 msgr2=0x7f6858112d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fd430 0x7f6858112d00 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f684c004070 tx=0x7f684c009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 shutdown_connections 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f684006d0e0 0x7f684006f590 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f68580fc0a0 0x7f68581107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 --2- 172.21.15.77:0/3907213526 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f68580fd430 0x7f6858112d00 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 >> 172.21.15.77:0/3907213526 conn(0x7f68580f6ca0 msgr2=0x7f68581002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 shutdown_connections 2024-02-19T19:03:55.290 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.270+0000 7f685cecb700 1 -- 172.21.15.77:0/3907213526 wait complete. 2024-02-19T19:03:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.373+0000 7f9263fff700 1 -- 172.21.15.77:0/1385107649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fce70 msgr2=0x7f92640fd2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:55.394 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.373+0000 7f9263fff700 1 --2- 172.21.15.77:0/1385107649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fce70 0x7f92640fd2c0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f9254004940 tx=0x7f9254009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.373+0000 7f9263fff700 1 -- 172.21.15.77:0/1385107649 shutdown_connections 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.373+0000 7f9263fff700 1 --2- 172.21.15.77:0/1385107649 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fce70 0x7f92640fd2c0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.373+0000 7f9263fff700 1 --2- 172.21.15.77:0/1385107649 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92640fbc60 0x7f92640fc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.373+0000 7f9263fff700 1 -- 172.21.15.77:0/1385107649 >> 172.21.15.77:0/1385107649 conn(0x7f92640f7300 msgr2=0x7f92640f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.373+0000 7f9263fff700 1 -- 172.21.15.77:0/1385107649 shutdown_connections 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.373+0000 7f9263fff700 1 -- 172.21.15.77:0/1385107649 wait complete. 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9263fff700 1 Processor -- start 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9263fff700 1 -- start start 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9263fff700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fbc60 0x7f926410c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9263fff700 1 --2- >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92640fce70 0x7f926410cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9263fff700 1 -- --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926410d2b0 con 0x7f92640fbc60 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9263fff700 1 -- --> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926410daf0 con 0x7f92640fce70 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9262ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fbc60 0x7f926410c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9262ffd700 1 --2- >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fbc60 0x7f926410c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.77:3300/0 says I am v2:172.21.15.77:49470/0 (socket says 172.21.15.77:49470) 2024-02-19T19:03:55.395 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9262ffd700 1 -- 172.21.15.77:0/280534437 learned_addr learned my addr 172.21.15.77:0/280534437 (peer_addr_for_me v2:172.21.15.77:0/0) 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9262ffd700 1 -- 172.21.15.77:0/280534437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92640fce70 msgr2=0x7f926410cc20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9262ffd700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92640fce70 0x7f926410cc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9262ffd700 1 -- 172.21.15.77:0/280534437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9254005040 con 0x7f92640fbc60 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.374+0000 7f9262ffd700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fbc60 0x7f926410c6e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f925800b760 tx=0x7f92580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.375+0000 7f924b7fe700 1 -- 172.21.15.77:0/280534437 <== mon.0 v2:172.21.15.77:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f925800f6f0 con 0x7f92640fbc60 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.375+0000 7f924b7fe700 1 -- 172.21.15.77:0/280534437 <== mon.0 v2:172.21.15.77:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f925800fd20 con 0x7f92640fbc60 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.375+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926410ddd0 con 0x7f92640fbc60 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.375+0000 7f924b7fe700 1 -- 172.21.15.77:0/280534437 <== mon.0 v2:172.21.15.77:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9258011e40 con 0x7f92640fbc60 2024-02-19T19:03:55.396 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.375+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926410e390 con 0x7f92640fbc60 2024-02-19T19:03:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.376+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 --> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926404bed0 con 0x7f92640fbc60 2024-02-19T19:03:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.376+0000 7f924b7fe700 1 -- 172.21.15.77:0/280534437 <== mon.0 v2:172.21.15.77:3300/0 4 ==== mgrmap(e 31) v1 ==== 91207+0+0 (secure 0 0 0) 0x7f9258011670 con 0x7f92640fbc60 2024-02-19T19:03:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.376+0000 7f924b7fe700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f924c06d0e0 0x7f924c06f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-19T19:03:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.377+0000 7f924b7fe700 1 -- 172.21.15.77:0/280534437 <== mon.0 v2:172.21.15.77:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6066+0+0 (secure 0 0 0) 0x7f925808c120 con 0x7f92640fbc60 2024-02-19T19:03:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.377+0000 7f92627fc700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f924c06d0e0 0x7f924c06f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-19T19:03:55.397 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.377+0000 7f92627fc700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f924c06d0e0 0x7f924c06f590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9254004890 tx=0x7f9254009e30 comp rx=0 tx=0).ready entity=mgr.14714 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-19T19:03:55.400 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.381+0000 7f924b7fe700 1 -- 172.21.15.77:0/280534437 <== mon.0 v2:172.21.15.77:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181772 (secure 0 0 0) 0x7f9258059c30 con 0x7f92640fbc60 2024-02-19T19:03:55.611 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.591+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 --> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f92640008d0 con 0x7f924c06d0e0 2024-02-19T19:03:55.612 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.593+0000 7f924b7fe700 1 -- 172.21.15.77:0/280534437 <== mgr.14714 v2:172.21.15.77:6800/248682737 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+364 (secure 0 0 0) 0x7f92640008d0 con 0x7f924c06d0e0 2024-02-19T19:03:55.612 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-02-19T19:03:55.612 INFO:teuthology.orchestra.run.smithi077.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:20c75febd3cc6fee70b3b846b358905b187f9293f7c7c6707c7c369b69fd0526", 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout: "in_progress": true, 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout: "services_complete": [ 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout: "mgr" 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout: ], 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout: "progress": "2/2 daemons upgraded", 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout: "message": "Currently upgrading grafana daemons", 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout: "is_paused": false 2024-02-19T19:03:55.613 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f924c06d0e0 msgr2=0x7f924c06f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f924c06d0e0 0x7f924c06f590 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9254004890 tx=0x7f9254009e30 comp rx=0 tx=0).stop 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fbc60 msgr2=0x7f926410c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fbc60 0x7f926410c6e0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f925800b760 tx=0x7f92580091a0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 shutdown_connections 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:6800/248682737,v1:172.21.15.77:6801/248682737] conn(0x7f924c06d0e0 0x7f924c06f590 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] conn(0x7f92640fbc60 0x7f926410c6e0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 --2- 172.21.15.77:0/280534437 >> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] conn(0x7f92640fce70 0x7f926410cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 >> 172.21.15.77:0/280534437 conn(0x7f92640f7300 msgr2=0x7f92641000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-19T19:03:55.615 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 shutdown_connections 2024-02-19T19:03:55.616 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:03:55.595+0000 7f9263fff700 1 -- 172.21.15.77:0/280534437 wait complete. 2024-02-19T19:04:09.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:08.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:09.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:08.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:09.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:08.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:10.310 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:09.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:10.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:09.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:10.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:09.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:10.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:10.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:10.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:10.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:10.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:10.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:11.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:10.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:11.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:10.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:11.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:10.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:11.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:11.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:11.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:11.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:11.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:11.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:12.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:11.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:12.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:11.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:12.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:11.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:12.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:11.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:12.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:11.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:12.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:11.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:12.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:12.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:12.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:12.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:12.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:12.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:13.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:12.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:13.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:12.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:13.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:12.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:13.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:12.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:13.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:12.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:13.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:12.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:13.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:13.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:13.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:13.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:13.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:13.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:14.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:13.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:14.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:13.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:14.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:13.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:14.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:13.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:14.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:13.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:14.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:13.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:14.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:14.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:14.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:14.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:14.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:14.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:15.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:14.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:15.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:14.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:15.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:14.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:15.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:14.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:15.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:14.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:15.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:14.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:15.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:15.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:15.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:15.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:15.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:15.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:16.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:15.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:16.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:15.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:16.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:15.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:16.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:15.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:16.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:15.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:16.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:15.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:16.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:16.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:16.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:16.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:16.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:16.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:17.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:16.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:17.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:16.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:17.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:16.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:17.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:16.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:17.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:16.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:17.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:16.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:17.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:17.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:17.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:17.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:17.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:17.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:18.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:17.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:18.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:17.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:18.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:17.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:18.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:17.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:18.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:17.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:18.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:17.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:18.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:18.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:18.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:18.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:18.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:18.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:19.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:18.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:19.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:18.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:19.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:18.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:19.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:18.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:19.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:18.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:19.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:18.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:19.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:19.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:19.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:19.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:19.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:19.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:20.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:19.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:20.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:19.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:20.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:19.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:20.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:19.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:20.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:19.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:20.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:19.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:20.848 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:20.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:20.848 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:20.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:20.848 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:20.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:20.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:20.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:20.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:20.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:20.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:20.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:21.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:20.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:21.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:20.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:21.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:20.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:21.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:21.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:21.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:21.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:21.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:21.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:21.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:21.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:21.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:21.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:21.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:21.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:22.310 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:21.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:22.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:21.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:22.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:21.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:22.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:22.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:22.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:22.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:22.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:22.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:22.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:22.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:22.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:22.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:22.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:22.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:22.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:22.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:22.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:22.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:22.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:22.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:23.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:23.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:23.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:23.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:23.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:23.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:23.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:23.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:23.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:23.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:23.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:23.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:24.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:23.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:24.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:23.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:24.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:23.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:24.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:24.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:24.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:24.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:24.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:24.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:24.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:24.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:24.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:24.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:24.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:24.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:25.310 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:24.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:25.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:24.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:25.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:24.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:26.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:25.795+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:26.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:25.795+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:26.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:25.795+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:26.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:25.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:26.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:25.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:26.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:25.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:26.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:25.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:26.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:25.547+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:26.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:25.547+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:27.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:26.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:27.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:26.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:27.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:26.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:27.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:26.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:27.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:26.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:27.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:26.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:27.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:26.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:27.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:26.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:27.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:26.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:27.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:27.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:27.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:27.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:27.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:27.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:27.973 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:27.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:27.973 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:27.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:27.973 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:27.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:27.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:27.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:27.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:27.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:27.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:27.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:28.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:28.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:28.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:28.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:28.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:28.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:28.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:28.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:28.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:28.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:28.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:28.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:29.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:28.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:29.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:28.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:29.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:28.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:29.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:29.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:29.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:29.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:29.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:29.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:29.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:29.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:29.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:30.310 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:29.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:30.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:29.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:30.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:29.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:31.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:30.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:31.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:30.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:31.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:30.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:31.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:30.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:31.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:30.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:31.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:30.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:31.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:30.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:31.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:30.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:31.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:30.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:32.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:31.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:32.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:31.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:32.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:31.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:32.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:31.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:32.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:31.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:32.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:31.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:32.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:31.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:32.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:31.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:32.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:31.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:32.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:32.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:32.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:32.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:32.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:32.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:32.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:32.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:32.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:32.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:32.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:32.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:32.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:32.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:32.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:32.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:32.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:32.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:33.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:33.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:33.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:33.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:33.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:33.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:33.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:33.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:33.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:33.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:33.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:33.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:34.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:33.785+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:34.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:33.785+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:34.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:33.785+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:35.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:34.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:35.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:34.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:35.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:34.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:35.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:34.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:35.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:34.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:35.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:34.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:35.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:34.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:35.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:34.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:35.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:34.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:36.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:35.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:36.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:35.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:36.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:35.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:36.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:35.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:36.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:35.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:36.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:35.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:35.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:35.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:35.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:37.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:36.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:37.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:36.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:37.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:36.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:37.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:36.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:37.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:36.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:37.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:36.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:37.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:36.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:37.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:36.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:37.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:36.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:37.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:37.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:37.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:37.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:37.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:37.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:37.970 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:37.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:37.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:37.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:37.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:37.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:37.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:37.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:37.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:37.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:37.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:37.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:39.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:38.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:39.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:38.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:39.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:38.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:39.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:38.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:39.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:38.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:39.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:38.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:39.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:38.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:39.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:38.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:39.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:38.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:39.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:39.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:39.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:39.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:39.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:39.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:39.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:39.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:40.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:39.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:40.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:39.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:40.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:39.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:41.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:40.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:41.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:40.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:41.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:40.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:41.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:40.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:41.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:40.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:41.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:40.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:41.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:40.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:41.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:40.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:41.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:40.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:42.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:41.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:42.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:41.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:42.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:41.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:42.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:41.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:42.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:41.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:42.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:41.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:42.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:41.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:42.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:41.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:42.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:41.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:42.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:42.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:42.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:42.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:42.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:42.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:42.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:42.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:42.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:42.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:42.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:42.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:42.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:42.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:42.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:42.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:42.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:42.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:43.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:43.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:43.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:43.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:43.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:43.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:43.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:43.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:43.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:43.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:43.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:43.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:45.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:44.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:45.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:44.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:45.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:44.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:45.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:44.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:45.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:44.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:45.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:44.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:44.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:44.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:44.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:45.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:45.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:45.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:45.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:45.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:45.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:45.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:45.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:45.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:45.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:45.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:45.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:45.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:45.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:45.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:45.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:45.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:45.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:46.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:46.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:46.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:46.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:46.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:46.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:46.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:46.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:46.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:46.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:46.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:46.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:46.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:46.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:46.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:46.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:46.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:46.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:47.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:47.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:47.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:47.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:47.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:47.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:47.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:47.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:47.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:47.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:47.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:47.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:48.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:47.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:48.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:47.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:48.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:47.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:48.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:48.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:48.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:48.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:48.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:48.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:48.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:48.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:48.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:48.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:48.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:48.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:48.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:48.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:48.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:48.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:48.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:48.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:49.594 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:49.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:49.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:49.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:49.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:49.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:50.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:49.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:50.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:49.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:50.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:49.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:50.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:49.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:50.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:49.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:50.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:49.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:50.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:50.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:50.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:50.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:50.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:50.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:50.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:50.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:50.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:50.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:50.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:50.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:50.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:50.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:50.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:50.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:50.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:50.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:51.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:51.734+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:51.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:51.734+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:51.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:51.734+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:51.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:51.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:51.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:51.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:51.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:51.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:51.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:51.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:51.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:51.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:51.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:51.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:52.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:52.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:52.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:52.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:52.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:52.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:52.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:52.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:52.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:52.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:52.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:52.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:52.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:52.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:52.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:52.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:52.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:52.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:53.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:53.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:53.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:53.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:53.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:53.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:53.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:53.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:53.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:53.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:53.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:53.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:54.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:53.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:54.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:53.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:54.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:53.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:54.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:54.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:54.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:54.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:54.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:54.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:54.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:54.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:54.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:54.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:54.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:54.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:54.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:54.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:54.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:54.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:54.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:54.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:56.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:55.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:56.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:55.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:56.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:55.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:56.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:55.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:56.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:55.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:56.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:55.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:56.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:55.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:56.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:55.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:56.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:55.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:57.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:56.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:57.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:56.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:57.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:56.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:57.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:56.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:57.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:56.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:57.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:56.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:57.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:56.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:57.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:56.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:57.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:56.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:57.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:57.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:57.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:57.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:57.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:57.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:57.973 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:57.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:57.973 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:57.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:57.973 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:57.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:57.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:57.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:57.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:57.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:57.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:57.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:58.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:58.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:58.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:58.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:58.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:58.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:58.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:58.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:58.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:58.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:58.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:58.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:58.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:58.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:58.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:58.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:58.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:58.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:59.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:59.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:59.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:59.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:59.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:04:59.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:04:59.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:59.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:59.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:59.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:59.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:04:59.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:04:59.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:59.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:59.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:59.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:04:59.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:04:59.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:00.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:00.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:00.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:00.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:00.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:00.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:00.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:00.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:00.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:00.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:00.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:00.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:00.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:00.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:00.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:00.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:00.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:00.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:01.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:01.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:01.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:01.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:01.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:01.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:01.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:01.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:01.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:01.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:01.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:01.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:01.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:01.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:01.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:01.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:01.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:01.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:02.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:02.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:02.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:02.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:02.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:02.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:02.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:02.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:02.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:02.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:02.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:02.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:03.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:02.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:03.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:02.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:03.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:02.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:03.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:03.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:03.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:03.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:03.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:03.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:03.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:03.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:03.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:03.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:03.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:03.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:03.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:03.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:03.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:03.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:03.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:03.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:04.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:04.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:04.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:04.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:04.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:04.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:05.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:04.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:05.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:04.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:05.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:04.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:05.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:04.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:05.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:04.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:05.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:04.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:05.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:05.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:05.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:05.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:05.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:05.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:05.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:05.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:05.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:05.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:05.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:05.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:05.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:05.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:05.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:05.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:05.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:05.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:06.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:06.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:06.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:06.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:06.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:06.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:06.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:06.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:06.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:06.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:06.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:06.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:06.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:06.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:06.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:06.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:06.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:06.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:07.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:07.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:07.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:07.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:07.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:07.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:07.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:07.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:07.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:07.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:07.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:07.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:07.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:07.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:07.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:07.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:07.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:07.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:08.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:08.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:08.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:08.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:08.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:08.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:09.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:08.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:09.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:08.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:09.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:08.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:09.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:08.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:09.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:08.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:09.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:08.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:09.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:09.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:09.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:09.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:09.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:09.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:09.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:09.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:09.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:09.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:09.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:09.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:10.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:09.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:10.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:09.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:10.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:09.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:10.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:10.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:10.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:10.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:10.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:10.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:11.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:10.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:11.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:10.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:11.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:10.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:11.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:10.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:11.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:10.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:11.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:10.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:11.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:11.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:11.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:11.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:11.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:11.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:12.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:11.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:12.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:11.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:12.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:11.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:12.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:11.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:12.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:11.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:12.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:11.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:12.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:12.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:12.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:12.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:12.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:12.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:12.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:12.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:12.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:12.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:12.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:12.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:12.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:12.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:12.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:12.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:12.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:12.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:13.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:13.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:13.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:13.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:13.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:13.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:14.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:13.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:14.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:13.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:14.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:13.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:14.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:13.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:14.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:13.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:14.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:13.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:14.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:14.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:14.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:14.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:14.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:14.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:15.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:14.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:15.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:14.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:15.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:14.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:15.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:14.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:15.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:14.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:15.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:14.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:15.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:15.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:15.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:15.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:15.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:15.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:16.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:15.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:16.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:15.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:16.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:15.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:16.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:15.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:16.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:15.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:16.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:15.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:16.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:16.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:16.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:16.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:16.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:16.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:17.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:16.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:17.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:16.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:17.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:16.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:17.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:16.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:17.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:16.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:17.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:16.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:17.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:17.280+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:17.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:17.280+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:17.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:17.280+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:17.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:17.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:17.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:17.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:17.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:17.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:17.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:17.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:17.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:17.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:17.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:17.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:18.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:18.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:18.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:18.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:18.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:18.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:19.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:18.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:19.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:18.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:19.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:18.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:19.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:18.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:19.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:18.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:19.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:18.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:19.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:19.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:19.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:19.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:19.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:19.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:19.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:19.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:19.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:19.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:19.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:19.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:20.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:19.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:20.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:19.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:20.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:19.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:20.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:20.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:20.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:20.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:20.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:20.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:21.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:20.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:21.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:20.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:21.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:20.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:21.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:20.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:21.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:20.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:21.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:20.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:21.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:21.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:21.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:21.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:21.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:21.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:22.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:21.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:22.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:21.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:22.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:21.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:22.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:21.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:22.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:21.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:22.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:21.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:22.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:22.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:22.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:22.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:22.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:22.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:22.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:22.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:22.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:22.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:22.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:22.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:22.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:22.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:22.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:22.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:22.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:22.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:23.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:23.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:23.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:23.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:23.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:23.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:23.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:23.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:23.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:23.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:23.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:23.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:23.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:23.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:23.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:23.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:23.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:23.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:24.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:24.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:24.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:24.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:24.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:24.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:25.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:24.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:25.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:24.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:25.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:24.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:25.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:24.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:25.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:24.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:25.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:24.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:25.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:25.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:25.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:25.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:25.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:25.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:26.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:25.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:26.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:25.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:26.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:25.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:26.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:25.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:26.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:25.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:26.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:25.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:26.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:26.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:26.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:26.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:26.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:26.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:27.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:26.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:27.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:26.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:27.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:26.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:27.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:26.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:27.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:26.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:27.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:26.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:27.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:27.161+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:27.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:27.161+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:27.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:27.161+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:27.987 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:27.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:27.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:27.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:27.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:27.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:27.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:27.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:27.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:27.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:27.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:27.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:28.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:28.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:28.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:28.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:28.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:28.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:29.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:28.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:29.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:28.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:29.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:28.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:29.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:28.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:29.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:28.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:29.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:28.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:29.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:29.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:29.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:29.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:29.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:29.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:29.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:29.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:29.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:29.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:29.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:29.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:29.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:29.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:29.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:30.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:30.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:30.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:30.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:30.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:30.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:31.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:30.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:31.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:30.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:31.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:30.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:31.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:30.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:31.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:30.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:31.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:30.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:31.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:31.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:31.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:31.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:31.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:31.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:32.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:31.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:32.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:31.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:32.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:31.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:32.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:31.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:32.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:31.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:32.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:31.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:32.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:32.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:32.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:32.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:32.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:32.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:33.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:32.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:33.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:32.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:33.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:32.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:33.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:32.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:33.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:32.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:33.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:32.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:33.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:33.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:33.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:33.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:33.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:33.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:33.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:33.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:33.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:33.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:33.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:33.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:34.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:33.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:34.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:33.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:34.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:33.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:34.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:34.084+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:34.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:34.084+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:34.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:34.084+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:35.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:35.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:35.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:35.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:34.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:35.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:34.755+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:35.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:34.755+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:35.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:35.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:35.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:35.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:35.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:35.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:36.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:35.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:36.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:35.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:36.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:35.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:36.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:35.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:36.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:35.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:36.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:35.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:36.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:36.093+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:36.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:36.093+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:36.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:36.093+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:37.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:36.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:37.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:36.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:37.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:36.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:37.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:36.780+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:37.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:36.780+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:37.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:36.780+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:37.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:37.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:37.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:37.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:37.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:37.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:37.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:37.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:37.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:37.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:37.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:37.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:37.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:37.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:37.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:37.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:37.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:37.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:38.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:38.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:38.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:38.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:38.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:38.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:39.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:38.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:39.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:38.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:39.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:38.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:39.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:38.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:39.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:38.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:39.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:38.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:39.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:39.053+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:39.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:39.053+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:39.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:39.053+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:39.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:39.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:39.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:40.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:39.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:40.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:39.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:40.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:39.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:40.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:40.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:40.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:40.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:40.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:40.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:41.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:40.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:41.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:40.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:41.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:40.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:41.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:40.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:41.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:40.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:41.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:40.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:41.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:41.007+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:41.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:41.008+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:41.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:41.008+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:41.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:41.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:41.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:41.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:41.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:41.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:42.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:41.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:42.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:41.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:42.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:41.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:42.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:41.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:42.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:41.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:42.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:41.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:42.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:42.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:42.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:42.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:42.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:42.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:43.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:42.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:43.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:42.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:43.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:42.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:43.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:42.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:43.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:42.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:43.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:42.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:43.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:43.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:43.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:44.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:43.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:44.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:43.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:44.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:43.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:44.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:43.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:44.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:43.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:44.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:43.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:44.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:44.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:44.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:44.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:44.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:44.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:44.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:44.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:44.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:44.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:44.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:44.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:45.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:44.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:45.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:44.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:45.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:44.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:45.899 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:45.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:45.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:45.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:45.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:45.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:46.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:45.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:46.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:45.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:46.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:45.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:46.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:45.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:46.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:45.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:46.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:45.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:46.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:46.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:46.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:46.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:46.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:46.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:46.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:46.854+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:46.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:46.854+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:46.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:46.854+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:47.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:46.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:47.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:46.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:47.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:46.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:47.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:47.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:47.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:47.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:47.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:47.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:47.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:47.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:47.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:47.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:47.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:47.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:48.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:47.924+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:48.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:47.924+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:48.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:47.924+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:49.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:48.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:49.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:48.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:49.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:48.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:49.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:48.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:49.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:48.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:49.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:48.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:49.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:48.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:49.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:48.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:49.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:48.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:49.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:49.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:49.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:49.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:49.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:49.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:50.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:49.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:50.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:49.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:50.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:49.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:50.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:49.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:50.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:49.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:50.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:49.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:51.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:50.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:51.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:50.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:51.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:50.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:51.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:50.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:51.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:50.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:51.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:50.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:51.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:50.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:51.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:50.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:51.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:50.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:52.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:51.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:52.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:51.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:52.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:51.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:52.052 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:51.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:52.052 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:51.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:52.052 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:51.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:52.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:52.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:52.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:52.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:52.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:52.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:52.992 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:05:52.947+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:05:52.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:52.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:52.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:52.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:52.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:52.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:52.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:52.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:52.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:52.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:52.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:52.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:53.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:53.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:53.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:53.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:53.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:53.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:53.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:53.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:53.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:53.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:53.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:53.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:54.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:53.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:54.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:53.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:54.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:53.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:54.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:54.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:54.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:55.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:54.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:55.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:54.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:55.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:54.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:55.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:54.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:55.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:54.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:55.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:54.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:55.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:55.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:55.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:55.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:55.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:55.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:56.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:55.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:56.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:55.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:56.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:55.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:56.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:55.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:56.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:55.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:56.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:55.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:56.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:56.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:56.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:56.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:56.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:56.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:56.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:56.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:56.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:56.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:56.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:56.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:57.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:57.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:57.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:57.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:57.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:57.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:57.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:56.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:57.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:56.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:57.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:56.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:57.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:57.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:57.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:57.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:57.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:57.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:57.969 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:57.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:57.969 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:57.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:57.970 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:57.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:58.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:05:57.948+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:05:58.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:58.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:58.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:59.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:58.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:59.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:58.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:59.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:58.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:05:59.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:58.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:59.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:58.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:59.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:58.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:59.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:59.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:59.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:59.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:59.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:05:59.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:05:59.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:59.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:59.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:59.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:05:59.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:05:59.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:00.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:59.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:00.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:59.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:00.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:05:59.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:00.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:00.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:00.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:00.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:00.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:00.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:01.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:00.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:01.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:00.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:01.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:00.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:01.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:00.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:01.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:00.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:01.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:00.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:01.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:01.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:01.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:01.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:01.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:01.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:02.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:01.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:02.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:01.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:02.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:01.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:02.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:01.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:02.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:01.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:02.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:01.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:02.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:02.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:02.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:02.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:02.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:02.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:03.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:02.948+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:03.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:02.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:03.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:02.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:03.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:02.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:03.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:02.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:03.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:02.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:03.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:02.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:03.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:03.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:03.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:03.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:03.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:03.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:04.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:03.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:04.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:03.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:04.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:03.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:04.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:03.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:04.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:03.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:04.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:03.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:04.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:04.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:04.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:04.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:04.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:04.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:05.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:04.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:05.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:04.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:05.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:04.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:05.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:04.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:05.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:04.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:05.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:04.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:05.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:05.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:05.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:05.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:05.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:05.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:06.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:05.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:06.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:05.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:06.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:05.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:06.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:05.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:06.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:05.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:06.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:05.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:06.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:06.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:06.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:06.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:06.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:06.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:07.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:06.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:07.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:06.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:07.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:06.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:07.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:06.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:07.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:06.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:07.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:06.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:07.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:07.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:07.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:07.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:07.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:07.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:08.213 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:07.949+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:08.214 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:07.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:08.214 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:07.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:08.214 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:07.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:08.214 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:08.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:08.214 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:08.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:08.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:08.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:08.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:07.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:08.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:07.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:08.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:07.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:08.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:08.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:08.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:08.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:08.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:08.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:09.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:08.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:09.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:08.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:09.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:08.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:09.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:09.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:09.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:09.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:09.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:09.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:10.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:09.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:10.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:09.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:10.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:09.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:10.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:10.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:10.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:10.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:10.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:10.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:10.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:09.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:10.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:09.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:10.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:09.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:11.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:10.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:11.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:10.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:11.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:10.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:11.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:10.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:11.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:10.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:11.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:10.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:11.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:11.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:11.314 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:11.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:11.314 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:11.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:12.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:11.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:12.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:11.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:12.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:11.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:12.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:11.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:12.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:11.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:12.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:11.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:12.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:12.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:12.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:12.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:12.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:12.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:13.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:12.949+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 87 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:13.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:12.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:13.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:12.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:13.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:12.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:13.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:12.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:13.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:12.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:13.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:12.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:13.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:13.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:13.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:13.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:13.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:13.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:14.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:13.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:14.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:13.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:14.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:13.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:14.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:13.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:14.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:13.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:14.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:13.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:14.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:14.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:14.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:14.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:14.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:14.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:15.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:14.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:15.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:14.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:15.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:14.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:15.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:14.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:15.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:14.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:15.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:14.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:15.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:15.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:15.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:15.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:15.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:15.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:16.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:15.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:16.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:15.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:16.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:15.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:16.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:15.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:16.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:15.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:16.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:15.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:16.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:16.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:16.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:16.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:16.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:16.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:17.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:16.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:17.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:16.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:17.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:16.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:17.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:17.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:17.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:17.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:17.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:17.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:17.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:16.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:17.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:16.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:17.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:16.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:18.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:17.950+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 119 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:18.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:17.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:18.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:17.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:18.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:17.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:18.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:18.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:18.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:18.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:18.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:18.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:18.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:18.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:18.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:18.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:18.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:18.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:19.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:18.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:19.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:18.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:19.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:18.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:19.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:19.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:19.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:19.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:19.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:19.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:19.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:19.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:19.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:19.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:19.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:19.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:20.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:20.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:20.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:20.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:20.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:20.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:20.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:20.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:20.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:20.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:20.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:20.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:20.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:19.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:20.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:19.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:20.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:19.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:21.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:20.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:21.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:20.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:21.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:20.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:21.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:21.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:21.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:21.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:21.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:21.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:21.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:20.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:21.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:20.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:21.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:20.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:22.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:21.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:22.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:21.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:22.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:21.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:22.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:22.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:22.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:22.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:22.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:22.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:22.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:21.985+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:22.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:21.985+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:22.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:21.985+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:23.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:22.951+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 147 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:23.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:22.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:23.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:22.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:23.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:22.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:23.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:22.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:23.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:22.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:23.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:22.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:23.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:23.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:23.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:23.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:23.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:23.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:24.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:24.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:24.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:24.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:24.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:24.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:24.113 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:23.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:24.113 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:23.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:24.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:23.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:24.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:24.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:24.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:24.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:24.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:24.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:25.163 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:25.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:25.163 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:25.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:25.163 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:25.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:25.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:24.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:25.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:24.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:25.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:24.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:25.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:25.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:25.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:25.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:26.157 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:26.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:26.158 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:26.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:26.158 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:26.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:26.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:25.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:26.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:25.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:26.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:25.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:26.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:26.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:26.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:26.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:26.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:26.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:27.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:26.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:27.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:26.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:27.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:26.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:27.165 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:26.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:27.165 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:26.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:27.165 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:26.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:27.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:27.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:27.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:27.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:27.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:28.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:27.951+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 180 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:28.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:27.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:28.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:27.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:28.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:27.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:28.195 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:27.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:28.195 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:27.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:28.195 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:27.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:28.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:28.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:28.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:28.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:28.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:28.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:29.240 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:28.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:29.241 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:28.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:29.241 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:28.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:29.241 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:28.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:29.241 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:28.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:29.241 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:28.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:29.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:29.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:29.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:30.214 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:29.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:30.214 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:29.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:30.214 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:29.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:30.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:29.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:30.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:29.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:30.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:29.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:30.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:30.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:30.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:30.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:30.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:30.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:31.246 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:30.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:31.247 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:30.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:31.247 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:30.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:31.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:30.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:31.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:30.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:31.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:30.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:31.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:31.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:31.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:31.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:31.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:31.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:32.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:31.869+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:32.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:31.869+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:32.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:31.869+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:32.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:31.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:32.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:31.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:32.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:31.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:32.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:32.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:32.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:32.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:32.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:32.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:33.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:32.952+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 212 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:33.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:32.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:33.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:32.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:33.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:32.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:33.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:32.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:33.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:32.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:33.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:32.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:33.381 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:33.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:33.381 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:33.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:33.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:33.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:34.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:33.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:34.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:33.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:34.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:33.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:34.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:33.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:34.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:33.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:34.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:33.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:34.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:34.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:34.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:34.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:34.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:34.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:35.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:34.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:35.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:34.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:35.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:34.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:35.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:34.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:35.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:34.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:35.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:34.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:35.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:35.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:35.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:35.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:35.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:35.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:36.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:35.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:36.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:35.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:36.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:35.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:36.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:35.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:36.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:35.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:36.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:35.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:36.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:36.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:36.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:36.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:36.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:36.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:37.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:36.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:37.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:36.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:37.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:36.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:37.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:36.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:37.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:36.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:37.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:36.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:37.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:37.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:37.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:37.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:37.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:37.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:38.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:37.953+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 240 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:38.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:37.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:38.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:37.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:38.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:37.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:38.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:37.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:38.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:37.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:38.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:37.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:38.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:38.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:38.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:38.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:38.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:38.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:39.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:38.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:39.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:38.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:39.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:38.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:39.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:38.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:39.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:38.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:39.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:38.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:39.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:39.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:39.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:39.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:39.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:39.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:40.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:39.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:40.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:39.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:40.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:39.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:40.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:39.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:40.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:39.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:40.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:39.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:40.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:40.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:40.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:40.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:41.256 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:40.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:41.256 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:40.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:41.256 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:40.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:41.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:40.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:41.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:40.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:41.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:40.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:41.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:41.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:41.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:41.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:41.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:41.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:42.264 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:41.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:42.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:41.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:42.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:41.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:42.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:41.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:42.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:41.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:42.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:41.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:42.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:42.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:42.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:42.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:42.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:42.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:42.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:42.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:42.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:42.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:42.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:42.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:43.255 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:42.953+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 272 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:43.255 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:42.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:43.255 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:42.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:43.255 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:42.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:43.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:43.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:43.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:43.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:43.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:43.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:43.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:43.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:43.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:44.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:43.967+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:44.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:43.967+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:44.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:43.967+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:44.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:44.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:44.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:44.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:45.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:44.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:45.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:44.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:45.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:44.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:44.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:44.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:44.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:45.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:45.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:45.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:45.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:45.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:45.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:45.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:45.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:45.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:45.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:45.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:45.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:46.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:45.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:46.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:45.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:46.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:45.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:46.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:46.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:46.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:46.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:46.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:46.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:47.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:46.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:47.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:46.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:47.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:46.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:47.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:46.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:47.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:46.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:47.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:46.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:47.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:47.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:47.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:47.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:47.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:47.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:48.061 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:47.954+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 304 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:48.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:47.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:48.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:47.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:48.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:47.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:48.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:47.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:48.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:47.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:48.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:47.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:48.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:48.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:48.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:48.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:49.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:48.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:49.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:48.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:49.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:48.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:49.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:49.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:49.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:49.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:49.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:49.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:49.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:49.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:49.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:49.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:49.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:49.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:50.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:49.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:50.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:49.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:50.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:49.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:50.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:50.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:50.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:50.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:50.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:50.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:50.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:50.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:50.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:50.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:50.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:50.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:51.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:50.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:51.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:50.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:51.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:50.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:51.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:51.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:51.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:51.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:51.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:51.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:51.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:51.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:51.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:51.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:51.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:51.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:52.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:51.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:52.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:51.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:52.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:51.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:52.329 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:52.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:52.330 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:52.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:52.330 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:52.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:52.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:52.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:52.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:52.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:52.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:52.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:52.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:52.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:52.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:52.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:52.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:52.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:53.311 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:52.955+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 342 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:53.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:53.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:53.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:53.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:53.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:53.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:53.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:53.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:53.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:53.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:53.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:53.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:54.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:53.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:54.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:53.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:54.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:53.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:54.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:53.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:53.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:53.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:54.719 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:54.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:54.719 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:54.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:54.719 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:54.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:54.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:54.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:54.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:54.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:54.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:54.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:55.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:54.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:55.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:54.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:55.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:54.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:55.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:55.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:55.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:55.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:55.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:56.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:55.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:56.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:55.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:56.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:55.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:56.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:56.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:56.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:56.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:56.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:56.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:56.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:56.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:56.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:56.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:56.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:56.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:56.995 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:56.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:56.996 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:56.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:56.996 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:56.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:57.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:56.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:57.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:56.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:57.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:56.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:57.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:57.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:57.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:57.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:57.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:57.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:57.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:57.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:57.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:57.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:57.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:57.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:58.274 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:06:57.955+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 370 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:06:58.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:58.000+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:58.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:58.000+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:58.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:58.000+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:58.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:58.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:58.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:58.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:58.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:58.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:59.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:58.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:59.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:58.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:59.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:58.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:59.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:58.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:59.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:58.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:59.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:06:58.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:06:59.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:59.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:59.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:59.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:59.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:06:59.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:06:59.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:59.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:59.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:59.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:06:59.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:06:59.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:00.311 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:00.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:00.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:00.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:00.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:00.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:00.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:00.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:00.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:01.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:00.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:01.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:00.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:01.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:00.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:01.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:00.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:01.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:00.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:01.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:00.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:01.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:01.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:01.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:02.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:01.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:02.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:01.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:02.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:01.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:02.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:01.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:02.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:01.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:02.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:01.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:02.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:02.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:02.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:03.105 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:02.956+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 398 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:03.105 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:02.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:03.105 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:02.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:03.105 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:02.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:03.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:02.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:03.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:02.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:03.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:02.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:03.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:03.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:03.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:04.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:03.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:04.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:03.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:04.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:03.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:04.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:03.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:04.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:03.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:04.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:03.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:04.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:04.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:04.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:05.264 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:04.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:05.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:04.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:05.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:04.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:05.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:04.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:05.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:04.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:05.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:04.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:05.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:05.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:05.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:05.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:06.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:05.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:06.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:05.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:06.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:05.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:06.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:05.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:06.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:05.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:06.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:05.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:06.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:06.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:06.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:06.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:06.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:06.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:07.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:06.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:07.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:06.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:07.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:06.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:07.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:06.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:07.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:06.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:07.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:06.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:07.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:06.936+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.14578.0:9243 3.8 3:13ccead7:::100000050b6.00000000:head [write 0~32897 [1@-1] in=32897b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:07.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:07.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:07.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:07.274+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:08.312 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:07.957+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 426 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:08.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:07.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:08.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:07.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:08.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:07.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:08.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:07.836+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:08.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:07.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:08.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:07.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:08.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:07.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:08.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:07.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:08.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:08.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:08.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:08.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:08.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:08.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:08.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:08.305+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:09.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:08.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:09.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:08.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:09.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:08.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:09.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:08.869+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:09.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:08.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:09.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:08.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:09.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:08.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:09.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:08.862+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:09.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:09.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:09.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:09.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:09.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:09.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:09.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:09.270+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:10.248 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:09.908+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:10.248 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:09.908+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:10.248 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:09.908+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:10.248 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:09.908+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:10.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:09.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:10.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:09.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:10.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:09.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:10.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:09.858+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:10.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:10.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:10.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:10.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:10.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:10.228+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:11.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:10.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:11.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:10.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:11.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:10.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:11.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:10.864+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:11.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:10.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:11.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:10.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:11.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:10.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:11.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:10.881+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:11.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:11.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:11.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:11.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:11.181+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:12.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:11.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:12.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:11.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:12.206 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:11.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:12.206 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:11.932+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:12.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:11.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:12.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:11.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:12.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:11.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:12.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:11.835+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:12.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:12.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:12.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:12.185+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:13.197 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:12.958+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 454 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:13.197 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:12.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:13.197 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:12.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:13.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:12.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:13.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:12.900+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:13.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:12.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:13.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:12.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:13.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:12.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:13.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:12.844+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:13.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:13.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:13.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:13.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:13.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:13.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:13.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:13.177+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:14.223 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:13.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:14.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:13.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:14.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:13.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:14.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:13.925+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:14.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:13.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:14.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:13.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:14.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:13.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:14.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:13.847+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:14.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:14.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:14.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:14.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:14.203+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:15.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:14.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:15.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:14.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:15.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:14.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:15.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:14.951+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:15.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:14.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:15.178 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:14.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:15.178 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:14.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:15.178 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:14.848+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:15.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:15.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:15.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:15.157+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:16.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:15.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:16.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:15.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:16.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:15.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:16.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:15.976+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:16.172 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:15.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:16.172 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:15.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:16.172 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:15.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:16.172 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:15.818+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:16.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:16.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:16.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:16.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:16.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:16.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:16.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:16.151+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:17.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:16.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:17.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:16.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:17.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:16.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:17.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:16.983+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:17.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:16.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:17.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:16.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:17.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:16.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:17.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:16.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:17.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:17.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:17.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:17.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:17.188+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:18.104 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:17.958+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 482 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:18.104 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:17.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:18.104 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:17.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:18.104 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:17.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:18.104 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:17.959+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:18.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:17.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:18.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:17.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:18.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:17.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:18.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:17.813+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:18.155+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:18.155+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:18.155+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:18.156+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:19.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:18.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:19.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:18.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:19.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:18.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:19.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:18.765+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:19.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:18.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:19.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:18.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:19.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:18.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:19.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:18.937+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:19.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:19.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:19.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:19.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:19.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:19.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:19.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:19.201+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:20.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:19.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:20.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:19.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:20.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:19.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:20.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:19.900+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:20.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:19.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:20.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:19.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:20.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:19.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:20.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:19.786+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:20.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:20.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:20.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:20.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:21.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:20.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:21.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:20.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:21.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:20.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:21.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:20.898+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:21.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:20.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:21.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:20.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:21.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:20.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:21.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:20.803+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:21.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:21.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:21.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:21.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:22.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:21.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:22.194 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:21.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:22.194 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:21.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:22.194 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:21.854+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:22.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:21.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:22.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:21.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:22.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:21.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:22.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:21.836+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:22.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:22.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:22.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:22.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:22.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:23.103 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:22.959+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 510 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:23.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:22.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:23.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:22.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:23.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:22.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:23.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:22.803+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:23.104 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:22.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:23.104 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:22.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:23.104 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:22.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:23.104 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:22.891+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:23.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:23.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:23.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:23.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:23.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:23.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:23.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:23.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:23.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:23.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:23.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:23.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:23.768+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:24.236 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:23.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:24.236 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:23.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:24.237 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:23.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:24.237 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:23.909+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:24.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:24.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:24.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:24.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:24.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:24.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:24.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:24.216+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:25.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:24.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:25.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:24.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:25.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:24.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:25.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:24.729+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:25.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:24.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:25.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:24.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:25.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:24.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:25.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:24.893+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:25.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:25.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:25.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:25.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:25.185+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:25.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:25.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:25.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:25.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:25.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:25.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:25.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:25.750+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:26.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:25.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:26.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:25.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:26.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:25.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:26.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:25.922+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:26.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:26.155+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:26.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:26.155+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:26.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:26.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:26.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:26.156+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:27.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:26.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:27.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:26.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:27.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:26.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:27.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:26.950+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:27.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:26.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:27.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:26.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:27.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:26.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:27.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:26.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:27.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:27.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:27.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:27.163+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:28.062 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:27.959+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 538 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:28.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:27.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:28.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:27.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:28.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:27.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:28.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:27.955+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:28.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:27.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:28.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:27.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:28.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:27.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:28.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:27.768+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:28.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:28.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:28.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:28.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:28.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:28.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:28.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:28.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:28.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:28.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:28.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:28.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:28.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:28.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:28.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:28.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:29.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:28.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:29.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:28.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:29.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:28.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:29.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:28.937+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:29.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:29.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:29.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:29.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:29.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:29.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:29.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:29.161+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:30.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:29.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:30.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:29.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:30.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:29.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:30.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:29.947+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:30.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:29.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:30.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:29.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:30.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:29.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:30.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:29.793+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:30.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:30.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:30.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:30.182+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:31.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:30.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:31.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:30.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:31.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:30.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:31.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:30.788+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:31.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:30.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:31.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:30.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:31.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:30.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:31.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:30.973+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:31.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:31.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:31.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:31.181+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:31.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:31.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:31.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:31.807+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:32.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:32.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:32.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:32.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:32.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:32.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:32.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:32.021+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:32.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:32.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:32.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:32.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:32.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:32.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:32.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:32.208+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:33.109 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:32.960+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 566 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:33.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:32.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:33.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:32.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:33.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:32.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:33.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:32.997+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:33.110 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:32.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:33.110 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:32.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:33.110 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:32.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:33.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:32.796+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:33.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:33.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:33.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:33.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:33.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:33.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:33.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:33.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:34.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:33.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:34.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:33.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:34.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:33.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:34.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:33.796+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:34.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:33.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:34.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:33.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:34.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:33.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:34.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:33.993+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:34.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:34.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:34.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:34.254+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:34.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:34.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:34.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:34.764+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:35.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:35.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:35.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:35.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:35.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:35.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:35.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:35.250+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:35.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:35.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:35.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:35.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:35.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:35.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:35.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:35.022+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:35.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:35.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:35.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:35.740+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:36.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:36.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:36.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:36.233+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:36.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:36.050+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:36.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:36.050+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:36.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:36.050+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:36.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:36.051+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:37.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:36.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:37.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:36.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:37.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:36.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:37.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:36.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:37.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:37.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:37.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:37.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:37.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:37.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:37.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:37.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:37.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:37.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:37.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:37.077+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:38.007 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:37.960+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 594 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:38.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:37.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:38.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:37.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:38.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:37.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:38.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:37.732+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:38.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:38.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:38.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:38.214+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:38.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:38.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:38.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:38.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:38.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:38.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:38.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:38.042+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:39.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:38.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:39.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:38.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:39.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:38.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:39.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:38.721+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:39.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:38.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:39.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:38.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:39.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:38.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:39.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:38.994+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:39.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:39.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:39.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:39.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:39.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:39.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:39.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:39.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:40.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:40.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:40.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:40.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:40.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:40.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:40.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:40.003+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:40.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:39.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:40.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:39.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:40.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:39.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:40.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:39.770+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:40.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:40.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:40.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:40.214+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:41.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:40.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:41.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:40.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:41.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:40.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:41.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:40.749+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:41.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:41.040+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:41.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:41.040+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:41.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:41.040+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:41.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:41.041+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:41.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:41.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:41.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:41.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:41.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:41.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:41.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:41.190+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:42.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:41.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:42.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:41.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:42.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:41.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:42.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:41.713+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:42.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:42.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:42.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:42.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:42.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:42.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:42.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:42.003+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:42.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:42.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:42.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:42.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:42.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:42.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:42.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:42.151+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:42.993 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:42.961+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 622 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:42.993 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:42.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:42.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:42.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:42.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:42.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:42.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:42.714+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:43.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:43.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:43.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:43.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:43.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:43.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:43.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:43.161+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:43.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:43.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:43.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:43.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:43.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:43.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:43.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:43.009+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:43.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:43.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:43.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:43.702+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:44.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:44.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:44.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:44.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:44.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:44.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:44.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:44.018+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:44.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:44.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:44.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:44.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:44.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:44.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:44.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:44.187+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:44.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:44.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:44.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:44.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:44.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:44.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:44.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:44.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:45.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:44.973+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:45.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:44.973+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:45.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:44.973+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:45.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:44.973+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:45.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:45.195+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:45.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:45.196+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:45.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:45.196+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:45.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:45.197+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:46.016 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:45.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:46.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:45.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:46.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:45.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:46.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:45.720+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:46.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:45.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:46.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:45.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:46.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:45.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:46.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:45.996+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:46.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:46.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:46.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:46.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:46.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:46.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:46.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:46.184+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:47.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:46.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:47.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:46.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:47.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:46.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:47.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:46.706+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:47.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:47.012+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:47.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:47.012+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:47.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:47.012+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:47.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:47.012+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:47.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:47.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:47.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:47.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:47.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:47.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:47.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:47.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:47.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:47.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:47.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:47.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:47.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:47.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:47.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:47.693+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:48.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:47.962+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 650 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:48.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:48.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:48.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:48.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:48.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:48.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:48.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:48.049+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:48.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:48.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:48.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:48.214+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:49.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:48.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:49.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:48.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:49.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:48.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:49.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:48.687+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:49.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:49.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:49.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:49.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:49.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:49.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:49.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:49.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:49.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:49.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:49.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:49.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:49.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:49.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:49.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:49.039+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:49.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:49.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:49.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:49.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:49.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:49.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:49.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:49.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:50.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:50.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:50.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:50.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:50.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:50.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:50.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:50.201+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:50.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:50.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:50.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:50.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:50.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:50.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:50.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:50.045+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:51.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:50.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:51.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:50.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:51.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:50.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:51.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:50.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:51.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:51.062+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:51.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:51.062+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:51.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:51.062+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:51.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:51.062+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:51.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:51.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:51.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:51.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:51.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:51.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:51.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:51.251+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:52.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:51.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:52.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:51.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:52.095 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:51.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:52.095 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:51.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:52.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:52.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:52.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:52.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:52.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:52.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:52.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:52.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:52.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:52.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:52.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:52.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:52.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:52.074+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:53.108 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:52.963+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 678 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:53.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:52.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:53.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:52.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:53.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:52.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:53.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:52.808+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:53.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:53.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:53.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:53.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:53.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:53.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:53.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:53.088+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:53.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:53.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:53.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:53.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:53.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:53.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:53.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:53.239+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:54.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:53.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:54.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:53.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:54.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:53.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:54.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:53.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:54.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:54.042+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:54.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:54.042+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:54.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:54.042+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:54.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:54.044+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:54.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:54.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:54.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:54.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:54.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:54.222+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:55.231 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:54.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:55.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:54.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:55.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:54.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:55.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:54.835+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:55.232 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:55.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:55.232 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:55.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:55.232 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:55.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:55.232 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:55.069+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:55.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:55.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:55.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:55.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:56.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:55.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:56.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:55.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:56.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:55.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:56.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:55.797+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:56.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:56.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:56.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:56.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:56.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:56.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:56.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:56.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:56.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:56.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:56.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:56.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:56.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:56.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:56.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:56.192+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:57.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:56.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:57.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:56.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:57.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:56.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:57.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:56.781+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:57.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:57.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:57.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:57.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:57.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:57.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:57.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:57.085+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:57.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:57.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:57.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:57.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:57.170+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:58.110 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:07:57.964+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 706 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:07:58.110 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:57.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:58.110 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:57.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:58.110 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:57.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:58.110 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:57.799+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:58.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:58.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:58.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:58.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:58.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:58.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:58.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:58.040+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:58.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:58.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:58.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:58.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:58.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:58.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:58.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:58.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:07:59.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:58.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:59.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:58.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:59.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:58.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:07:59.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:58.842+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:59.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:59.035+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:59.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:59.035+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:59.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:59.035+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:07:59.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:07:59.035+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:07:59.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:59.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:59.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:59.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:59.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:59.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:07:59.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:07:59.106+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:00.105 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:00.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:00.106 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:00.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:00.106 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:00.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:00.106 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:00.008+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:00.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:59.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:00.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:59.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:00.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:59.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:00.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:07:59.801+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:00.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:00.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:00.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:00.085+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:01.081 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:01.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:01.081 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:01.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:01.081 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:01.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:01.082 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:01.033+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:01.082 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:00.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:01.082 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:00.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:01.082 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:00.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:01.082 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:00.810+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:01.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:01.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:01.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:01.061+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:02.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:02.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:02.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:02.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:02.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:02.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:02.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:02.024+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:02.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:01.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:02.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:01.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:02.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:01.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:02.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:01.829+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:02.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:01.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:02.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:01.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:02.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:01.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:02.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:01.991+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:03.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:02.965+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 734 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:03.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:03.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:03.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:03.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:03.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:03.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:03.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:03.071+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:03.112 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:03.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:03.112 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:03.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:03.112 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:03.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:03.112 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:03.008+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:03.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:02.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:03.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:02.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:03.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:02.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:03.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:02.837+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:04.118 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:03.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:04.118 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:03.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:04.118 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:03.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:04.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:03.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:04.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:03.991+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:04.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:03.991+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:04.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:03.991+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:04.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:03.992+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:04.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:04.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:04.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:04.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:05.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:05.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:05.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:05.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:05.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:05.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:05.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:05.023+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:05.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:04.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:05.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:04.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:05.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:04.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:05.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:04.837+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:05.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:05.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:05.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:05.097+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:05.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:05.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:05.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:05.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:05.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:05.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:05.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:05.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:06.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:05.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:06.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:05.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:06.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:05.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:06.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:05.977+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:06.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:06.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:06.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:06.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:06.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:06.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:06.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:06.131+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:07.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:06.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:07.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:06.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:07.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:06.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:07.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:06.820+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:07.139 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:06.944+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:07.139 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:06.944+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:07.139 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:06.944+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:07.139 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:06.945+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:07.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:07.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:07.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:07.118+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:08.166 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:07.966+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 762 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:08.167 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:07.932+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:08.167 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:07.932+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:08.167 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:07.932+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:08.167 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:07.932+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:08.167 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:07.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:08.167 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:07.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:08.167 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:07.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:08.167 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:07.865+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:08.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:08.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:08.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:08.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:08.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:08.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:08.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:08.146+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:09.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:08.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:09.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:08.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:09.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:08.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:09.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:08.841+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:09.145 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:08.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:09.145 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:08.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:09.145 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:08.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:09.145 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:08.935+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:09.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:09.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:09.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:09.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:09.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:09.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:09.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:09.125+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:10.172 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:09.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:10.172 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:09.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:10.172 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:09.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:10.172 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:09.805+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:10.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:09.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:10.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:09.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:10.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:09.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:10.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:09.964+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:10.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:10.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:10.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:10.151+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:11.133 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:10.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:11.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:10.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:11.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:10.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:11.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:10.822+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:11.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:10.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:11.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:10.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:11.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:10.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:11.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:10.948+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:11.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:11.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:11.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:11.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:12.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:11.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:12.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:11.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:12.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:11.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:12.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:11.908+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:12.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:11.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:12.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:11.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:12.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:11.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:12.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:11.784+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:12.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:12.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:12.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:12.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:13.111 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:12.966+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 800 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:13.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:12.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:13.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:12.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:13.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:12.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:13.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:12.888+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:13.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:12.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:13.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:12.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:13.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:12.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:13.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:12.802+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:13.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:13.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:13.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:13.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:13.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:13.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:13.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:13.118+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:14.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:13.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:14.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:13.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:14.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:13.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:14.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:13.854+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:14.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:13.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:14.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:13.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:14.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:13.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:14.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:13.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:14.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:14.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:14.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:14.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:15.195 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:14.819+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:15.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:14.819+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:15.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:14.819+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:15.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:14.820+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:15.196 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:14.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:15.196 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:14.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:15.196 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:14.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:15.196 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:14.874+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:15.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:15.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:15.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:15.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:16.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:15.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:16.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:15.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:16.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:15.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:16.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:15.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:16.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:15.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:16.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:15.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:16.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:15.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:16.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:15.822+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:16.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:16.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:16.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:16.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:16.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:16.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:16.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:16.177+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:17.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:16.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:17.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:16.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:17.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:16.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:17.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:16.807+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:17.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:16.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:17.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:16.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:17.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:16.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:17.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:16.789+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:17.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:17.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:17.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:17.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:17.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:17.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:17.161+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:18.114 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:17.967+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 878 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:18.114 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:17.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:18.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:17.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:18.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:17.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:18.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:17.829+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:18.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:17.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:18.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:17.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:18.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:17.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:18.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:17.842+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:18.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:18.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:18.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:18.195+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:19.245 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:18.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:19.246 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:18.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:19.246 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:18.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:19.246 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:18.848+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:19.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:18.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:19.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:18.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:19.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:18.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:19.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:18.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:19.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:19.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:19.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:19.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:19.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:19.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:19.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:19.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:20.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:19.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:20.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:19.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:20.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:19.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:20.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:19.804+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:20.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:19.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:20.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:19.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:20.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:19.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:20.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:19.795+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:20.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:20.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:20.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:20.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:21.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:20.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:21.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:20.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:21.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:20.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:21.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:20.818+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:21.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:20.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:21.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:20.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:21.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:20.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:21.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:20.792+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:21.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:21.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:21.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:21.181+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:22.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:21.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:22.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:21.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:22.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:21.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:22.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:21.790+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:22.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:21.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:22.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:21.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:22.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:21.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:22.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:21.781+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:22.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:22.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:22.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:22.176+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:23.114 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:22.968+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 956 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:23.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:22.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:23.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:22.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:23.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:22.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:23.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:22.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:23.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:22.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:23.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:22.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:23.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:22.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:23.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:22.830+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:23.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:23.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:23.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:23.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:23.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:23.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:23.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:23.171+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:23.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:23.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:23.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:23.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:23.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:23.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:23.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:23.781+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:24.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:23.783+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:24.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:23.783+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:24.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:23.783+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:24.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:23.783+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:24.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:24.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:24.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:24.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:24.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:24.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:24.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:24.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:25.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:24.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:25.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:24.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:25.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:24.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:25.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:24.794+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:25.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:24.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:25.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:24.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:25.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:24.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:25.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:24.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:25.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:25.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:25.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:25.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:25.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:25.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:25.156+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:26.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:25.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:26.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:25.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:26.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:25.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:26.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:25.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:26.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:25.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:26.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:25.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:26.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:25.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:26.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:25.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:26.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:26.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:26.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:26.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:26.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:26.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:26.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:26.198+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:27.218 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:26.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:27.218 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:26.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:27.218 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:26.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:27.219 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:26.826+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:27.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:26.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:27.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:26.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:27.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:26.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:27.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:26.795+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:27.196+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:27.196+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:27.196+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:27.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:27.198+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:28.062 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:27.969+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1033 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:28.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:27.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:28.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:27.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:28.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:27.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:28.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:27.790+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:28.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:27.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:28.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:27.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:28.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:27.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:28.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:27.832+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:28.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:28.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:28.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:28.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:28.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:28.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:28.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:28.174+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:29.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:28.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:29.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:28.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:29.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:28.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:29.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:28.844+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:29.229 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:28.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:29.229 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:28.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:29.229 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:28.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:29.229 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:28.830+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:29.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:29.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:29.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:29.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:30.183 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:29.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:30.184 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:29.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:30.184 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:29.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:30.184 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:29.791+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:30.184 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:29.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:30.184 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:29.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:30.184 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:29.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:30.184 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:29.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:30.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:30.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:30.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:30.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:30.163+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:30.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:30.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:30.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:30.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:30.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:30.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:30.817 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:30.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:31.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:30.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:31.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:30.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:31.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:30.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:31.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:30.795+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:31.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:31.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:31.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:31.127+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:32.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:31.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:32.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:31.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:32.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:31.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:32.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:31.783+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:31.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:31.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:31.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:31.757+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:32.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:32.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:32.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:32.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:32.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:32.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:32.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:32.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:33.003 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:32.969+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1112 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:33.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:32.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:33.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:32.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:33.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:32.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:33.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:32.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:33.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:32.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:33.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:32.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:33.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:32.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:33.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:32.726+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:33.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:33.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:33.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:33.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:33.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:33.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:33.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:33.093+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:34.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:33.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:34.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:33.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:34.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:33.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:34.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:33.834+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:34.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:33.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:34.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:33.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:34.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:33.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:34.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:33.712+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:34.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:34.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:34.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:34.143+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:35.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:34.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:35.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:34.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:35.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:34.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:35.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:34.834+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:35.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:34.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:35.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:34.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:35.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:34.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:35.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:34.688+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:35.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:35.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:35.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:35.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:35.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:35.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:35.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:35.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:35.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:35.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:35.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:36.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:35.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:36.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:35.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:36.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:35.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:36.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:35.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:36.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:35.864+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:36.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:36.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:36.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:36.219+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:36.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:36.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:36.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:36.750+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:37.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:36.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:37.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:36.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:37.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:36.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:37.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:36.845+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:37.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:37.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:37.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:37.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:38.008 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:37.970+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1189 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:38.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:37.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:38.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:37.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:38.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:37.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:38.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:37.853+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:38.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:37.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:38.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:37.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:38.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:37.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:38.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:37.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:38.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:38.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:38.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:38.145+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:39.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:38.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:39.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:38.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:39.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:38.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:39.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:38.832+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:39.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:38.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:39.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:38.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:39.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:38.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:39.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:38.773+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:39.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:39.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:39.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:39.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:39.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:39.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:39.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:39.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:40.188 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:39.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:40.188 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:39.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:40.188 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:39.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:40.188 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:39.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:40.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:39.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:40.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:39.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:40.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:39.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:40.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:39.809+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:40.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:40.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:40.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:40.167+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:41.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:40.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:41.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:40.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:41.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:40.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:41.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:40.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:40.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:40.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:40.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:40.789+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:41.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:41.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:41.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:41.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:41.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:41.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:41.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:41.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:42.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:41.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:42.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:41.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:42.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:41.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:42.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:41.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:42.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:41.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:41.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:41.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:41.766+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:42.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:42.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:42.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:42.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:42.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:42.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:42.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:42.161+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:43.062 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:42.971+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1268 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:43.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:42.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:43.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:42.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:43.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:42.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:43.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:42.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:43.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:42.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:43.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:42.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:43.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:42.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:43.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:42.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:43.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:43.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:43.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:43.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:43.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:43.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:43.554 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:43.131+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:44.200 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:43.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:44.201 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:43.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:44.201 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:43.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:44.201 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:43.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:44.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:43.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:44.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:43.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:44.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:43.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:44.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:43.829+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:44.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:44.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:44.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:44.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:44.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:44.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:44.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:45.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:44.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:45.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:44.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:45.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:44.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:45.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:44.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:45.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:44.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:45.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:45.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:45.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:45.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:45.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:45.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:45.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:45.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:46.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:45.870+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:46.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:45.870+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:46.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:45.870+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:46.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:45.870+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:46.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:45.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:46.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:45.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:46.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:45.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:46.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:45.806+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:46.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:46.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:46.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:46.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:46.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:46.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:46.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:46.106+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:47.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:46.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:47.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:46.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:47.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:46.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:47.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:46.803+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:47.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:46.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:47.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:46.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:47.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:46.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:47.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:46.871+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:47.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:47.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:47.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:47.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:47.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:47.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:47.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:47.100+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:48.120 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:47.972+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1347 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:48.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:47.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:48.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:47.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:48.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:47.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:48.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:47.849+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:48.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:47.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:48.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:47.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:48.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:47.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:48.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:47.851+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:48.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:48.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:48.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:48.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:48.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:48.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:48.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:48.067+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:49.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:48.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:49.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:48.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:49.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:48.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:48.839+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:49.066+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:49.066+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:49.066+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:49.067+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:48.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:48.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:48.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:49.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:48.887+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:50.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:49.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:50.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:49.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:50.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:49.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:49.873+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:50.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:50.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:50.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:50.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:49.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:49.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:49.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:50.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:49.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:51.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:50.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:51.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:50.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:51.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:50.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:51.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:50.874+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:51.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:50.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:51.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:50.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:51.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:50.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:51.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:50.923+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:51.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:51.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:51.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:51.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:51.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:51.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:51.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:51.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:52.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:51.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:52.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:51.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:52.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:51.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:51.871+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:51.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:51.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:51.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:51.953+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:52.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:52.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:52.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:52.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:52.074+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:53.312 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:52.973+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1426 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:53.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:52.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:52.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:52.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:52.987+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:53.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:53.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:53.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:53.106+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:52.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:53.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:52.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:53.314 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:52.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:53.314 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:52.886+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:53.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:53.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:53.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:54.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:53.898+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:54.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:54.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:54.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:54.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:54.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:54.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:54.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:54.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:54.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:53.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:54.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:53.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:54.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:53.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:54.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:53.974+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:55.174 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:55.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:55.175 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:55.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:55.175 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:55.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:55.175 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:55.023+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:55.175 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:54.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:55.175 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:54.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:55.175 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:54.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:55.175 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:54.934+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:55.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:55.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:55.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:55.153+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:56.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:55.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:56.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:55.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:56.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:55.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:56.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:55.912+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:56.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:55.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:56.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:55.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:56.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:55.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:56.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:55.991+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:56.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:56.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:56.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:56.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:56.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:56.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:56.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:56.121+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:57.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:57.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:57.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:57.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:57.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:57.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:57.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:57.104+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:57.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:56.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:57.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:56.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:57.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:56.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:57.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:56.867+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:57.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:56.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:57.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:56.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:57.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:56.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:57.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:56.997+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:58.312 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:08:57.974+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1503 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:08:58.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:57.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:57.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:57.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:57.911+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:57.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:57.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:57.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:57.954+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:58.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:58.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:58.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:58.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:58.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:58.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:58.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:59.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:59.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:59.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:59.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:59.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:59.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:08:59.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:08:59.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:08:59.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:58.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:59.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:58.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:59.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:58.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:08:59.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:58.913+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:08:59.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:58.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:59.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:58.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:59.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:58.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:08:59.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:58.912+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:00.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:59.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:00.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:59.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:00.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:59.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:00.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:08:59.888+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:00.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:59.894+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:00.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:59.894+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:00.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:59.894+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:00.206 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:08:59.895+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:00.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:00.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:00.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:00.184+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:01.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:00.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:01.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:00.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:01.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:00.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:01.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:00.910+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:01.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:00.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:01.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:00.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:01.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:00.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:01.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:00.942+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:01.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:01.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:01.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:01.231+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:02.248 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:01.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:02.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:01.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:02.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:01.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:02.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:01.910+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:02.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:01.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:02.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:01.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:02.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:01.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:02.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:01.885+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:02.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:02.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:02.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:02.228+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:03.120 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:02.975+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1582 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:03.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:02.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:03.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:02.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:03.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:02.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:03.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:02.885+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:03.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:02.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:03.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:02.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:03.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:02.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:03.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:02.845+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:03.554 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:03.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:03.554 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:03.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:03.554 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:03.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:03.554 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:03.198+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:04.102 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:03.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:04.102 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:03.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:04.102 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:03.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:04.102 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:03.897+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:04.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:03.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:04.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:03.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:04.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:03.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:04.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:03.825+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:04.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:04.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:04.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:04.192+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:05.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:04.930+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:05.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:04.930+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:05.206 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:04.930+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:05.206 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:04.931+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:05.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:04.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:05.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:04.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:05.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:04.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:05.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:04.792+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:05.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:05.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:05.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:05.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:05.185+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:06.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:05.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:06.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:05.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:06.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:05.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:06.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:05.948+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:06.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:05.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:06.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:05.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:06.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:05.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:06.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:05.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:06.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:06.155+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:06.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:06.155+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:06.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:06.155+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:06.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:06.156+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:07.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:06.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:07.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:06.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:07.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:06.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:07.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:06.913+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:07.157 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:06.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:07.157 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:06.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:07.157 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:06.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:07.158 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:06.797+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:07.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:07.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:07.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:07.136+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:08.126 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:07.976+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1662 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:08.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:07.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:08.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:07.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:08.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:07.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:08.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:07.888+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:08.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:07.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:08.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:07.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:08.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:07.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:08.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:07.818+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:08.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:08.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:08.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:08.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:08.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:08.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:08.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:08.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:09.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:08.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:09.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:08.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:09.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:08.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:09.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:08.870+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:09.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:08.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:09.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:08.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:09.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:08.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:09.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:08.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:09.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:09.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:09.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:09.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:09.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:09.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:09.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:09.156+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:10.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:09.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:10.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:09.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:10.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:09.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:10.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:09.908+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:10.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:09.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:10.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:09.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:10.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:09.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:10.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:09.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:10.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:10.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:10.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:10.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:10.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:10.190+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:11.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:10.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:11.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:10.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:11.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:10.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:11.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:10.861+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:11.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:10.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:11.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:10.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:11.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:10.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:11.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:10.757+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:11.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:11.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:11.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:11.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:11.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:11.200+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:12.188 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:11.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:12.188 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:11.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:12.188 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:11.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:12.188 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:11.834+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:12.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:11.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:12.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:11.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:12.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:11.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:12.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:11.801+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:12.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:12.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:12.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:12.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:12.167+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:13.124 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:12.977+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1757 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:13.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:12.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:13.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:12.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:13.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:12.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:13.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:12.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:13.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:12.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:13.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:12.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:13.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:12.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:13.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:12.825+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:13.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:13.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:13.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:13.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:13.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:13.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:13.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:13.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:14.175 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:13.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:14.175 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:13.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:14.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:13.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:14.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:13.860+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:14.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:13.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:14.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:13.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:14.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:13.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:14.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:13.806+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:14.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:14.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:14.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:14.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:14.155+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:15.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:14.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:15.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:14.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:15.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:14.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:15.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:14.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:15.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:14.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:15.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:14.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:15.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:14.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:15.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:14.801+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:15.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:15.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:15.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:15.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:15.133+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:16.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:15.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:16.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:15.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:16.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:15.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:16.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:15.856+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:16.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:15.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:16.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:15.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:16.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:15.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:16.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:15.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:16.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:16.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:16.390 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:16.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:16.390 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:16.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:16.390 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:16.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:17.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:16.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:17.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:16.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:17.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:16.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:17.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:16.847+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:17.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:16.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:17.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:16.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:17.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:16.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:17.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:16.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:17.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:17.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:17.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:17.082+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:18.063 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:17.978+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1850 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:18.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:17.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:18.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:17.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:18.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:17.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:18.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:17.819+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:18.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:17.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:18.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:17.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:18.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:17.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:18.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:17.784+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:18.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:18.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:18.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:18.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:18.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:18.119+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:19.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:18.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:19.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:18.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:19.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:18.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:19.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:18.794+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:19.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:18.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:19.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:18.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:19.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:18.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:19.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:18.864+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:19.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:19.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:19.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:19.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:19.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:19.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:19.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:19.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:20.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:19.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:20.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:19.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:20.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:19.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:20.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:19.810+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:20.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:19.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:20.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:19.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:20.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:19.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:20.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:19.862+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:20.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:20.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:20.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:20.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:20.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:21.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:20.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:21.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:20.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:21.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:20.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:21.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:20.850+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:21.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:20.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:21.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:20.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:21.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:20.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:21.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:20.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:21.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:21.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:21.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:21.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:21.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:21.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:22.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:21.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:22.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:21.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:22.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:21.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:22.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:21.889+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:22.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:21.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:22.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:21.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:22.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:21.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:22.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:21.736+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:22.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:22.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:22.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:22.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:22.138+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:23.001 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:22.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:23.001 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:22.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:23.001 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:22.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:23.001 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:22.713+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:23.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:22.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:23.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:22.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:23.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:22.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:23.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:22.882+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:23.312 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:22.979+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 1945 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:23.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:23.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:23.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:23.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:23.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:23.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:23.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:23.151+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:23.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:23.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:23.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:23.726+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:24.186 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:23.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:24.186 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:23.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:24.187 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:23.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:24.187 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:23.870+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:24.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:24.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:24.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:24.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:24.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:24.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:24.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:24.166+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:25.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:24.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:25.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:24.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:25.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:24.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:25.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:24.843+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:25.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:24.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:25.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:24.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:25.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:24.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:25.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:24.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:25.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:25.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:25.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:25.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:25.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:25.730 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:09:25.708+0000 7f8323a29700 0 monclient(hunting): authenticate timed out after 300 2024-02-19T19:09:25.731 INFO:teuthology.orchestra.run.smithi077.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-19T19:09:25.863 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:25.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:25.863 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:25.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:25.863 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:25.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:25.864 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:25.719+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:26.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:25.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:26.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:25.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:26.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:25.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:26.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:25.842+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:26.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:26.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:26.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:26.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:26.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:26.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:26.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:26.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:26.442 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-02-19T19:09:26.705 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T19:09:27.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:26.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:27.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:26.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:27.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:26.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:27.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:26.687+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:27.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:26.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:27.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:26.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:27.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:26.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:27.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:26.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:27.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:27.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:27.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:27.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:27.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:27.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:27.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:27.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:28.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:27.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:28.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:27.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:28.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:27.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:28.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:27.798+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:28.002 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:27.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:28.002 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:27.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:28.002 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:27.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:28.002 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:27.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:28.252 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:27.979+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2038 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:28.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:28.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:28.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:28.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:28.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:28.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:28.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:28.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:29.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:28.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:29.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:28.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:29.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:28.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:29.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:28.756+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:29.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:28.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:29.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:28.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:29.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:28.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:29.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:28.764+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:29.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:29.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:29.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:29.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:29.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:29.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:29.111+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:30.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:29.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:30.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:29.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:30.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:29.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:30.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:29.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:30.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:29.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:30.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:29.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:30.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:29.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:30.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:29.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:30.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:30.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:30.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:30.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:30.148+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:31.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:30.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:31.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:30.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:31.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:30.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:31.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:30.801+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:31.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:30.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:31.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:30.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:31.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:30.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:31.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:30.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:31.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:31.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:31.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:31.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:31.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:31.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:31.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:31.181+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:32.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:31.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:31.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:31.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:31.795+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:32.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:31.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:32.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:31.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:32.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:31.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:32.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:31.798+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:32.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:32.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:32.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:32.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:32.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:32.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:32.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:32.232+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:33.125 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:32.980+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2133 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:33.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:32.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:33.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:32.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:33.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:32.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:33.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:32.849+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:33.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:32.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:33.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:32.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:33.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:32.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:33.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:32.809+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:33.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:33.195+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:33.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:33.195+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:33.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:33.195+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:33.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:33.196+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:34.181 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:33.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:34.181 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:33.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:34.181 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:33.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:34.181 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:33.835+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:34.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:33.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:34.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:33.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:34.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:33.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:34.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:33.826+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:34.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:34.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:34.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:34.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:34.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:34.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:34.160+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:35.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:34.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:35.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:34.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:35.225 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:34.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:35.225 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:34.809+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:35.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:34.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:35.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:34.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:35.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:34.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:35.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:34.805+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:35.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:35.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:35.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:35.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:35.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:35.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:35.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:35.203+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:36.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:35.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:36.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:35.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:36.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:35.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:36.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:35.857+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:36.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:35.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:36.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:35.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:36.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:35.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:36.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:35.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:36.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:36.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:36.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:36.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:36.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:36.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:36.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:36.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:37.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:36.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:37.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:36.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:37.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:36.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:37.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:36.772+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:36.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:36.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:36.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:36.865+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:37.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:37.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:37.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:37.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:37.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:37.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:37.264+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:38.027 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:37.982+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2226 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:38.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:37.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:38.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:37.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:38.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:37.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:38.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:37.753+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:38.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:37.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:38.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:37.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:38.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:37.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:38.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:37.821+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:38.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:38.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:38.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:38.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:38.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:38.242+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:39.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:38.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:39.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:38.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:39.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:38.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:39.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:38.791+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:39.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:38.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:39.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:38.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:39.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:38.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:39.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:38.775+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:39.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:39.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:39.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:39.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:39.312 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:39.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:39.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:39.246+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:39.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:39.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:39.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:39.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:39.745+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:40.270 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:39.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:40.271 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:39.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:40.271 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:39.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:40.271 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:39.837+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:40.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:40.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:40.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:40.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:40.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:40.249+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:41.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:40.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:41.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:40.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:41.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:40.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:41.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:40.859+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:40.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:40.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:40.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:40.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:41.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:41.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:41.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:41.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:41.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:41.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:41.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:41.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:42.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:41.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:42.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:41.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:42.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:41.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:42.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:41.839+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:41.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:41.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:41.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:41.758+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:42.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:42.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:42.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:42.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:42.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:42.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:42.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:42.246+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:43.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:42.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:43.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:42.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:43.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:42.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:43.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:42.884+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:43.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:42.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:43.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:42.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:43.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:42.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:43.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:42.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:43.264 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:42.982+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2320 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:43.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:43.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:43.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:43.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:43.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:43.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:43.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:43.243+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:43.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:43.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:43.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:43.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:43.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:43.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:44.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:43.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:44.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:43.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:44.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:43.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:44.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:43.910+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:44.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:44.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:44.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:44.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:44.276+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:44.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:44.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:44.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:44.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:45.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:44.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:45.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:44.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:45.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:44.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:45.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:44.921+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:45.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:45.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:45.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:45.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:45.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:45.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:45.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:45.270+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:45.976 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:45.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:45.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:45.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:45.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:45.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:45.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:45.614+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:46.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:45.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:46.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:45.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:46.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:45.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:46.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:45.955+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:46.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:46.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:46.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:46.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:46.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:46.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:46.665 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:46.294+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:46.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:46.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:46.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:46.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:46.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:46.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:46.954 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:46.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:47.238 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:46.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:47.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:46.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:47.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:46.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:47.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:46.932+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:47.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:47.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:47.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:47.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:47.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:47.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:47.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:47.262+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:47.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:47.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:48.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:47.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:48.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:47.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:48.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:47.648+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:48.254 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:47.983+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2414 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:48.254 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:47.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:48.254 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:47.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:48.254 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:47.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:48.254 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:47.978+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:48.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:48.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:48.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:48.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:48.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:48.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:48.233+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:49.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:48.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:49.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:48.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:49.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:48.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:49.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:48.623+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:49.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:48.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:49.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:48.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:49.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:48.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:49.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:48.986+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:49.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:49.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:49.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:49.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:49.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:49.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:49.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:49.268+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:50.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:49.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:50.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:49.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:50.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:49.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:50.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:49.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:50.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:50.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:50.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:50.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:50.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:50.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:50.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:50.002+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:50.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:50.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:50.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:50.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:50.261+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:51.020 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:50.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:51.020 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:50.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:51.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:50.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:51.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:50.625+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:51.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:50.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:51.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:50.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:51.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:50.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:51.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:50.999+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:51.620 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:51.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:51.620 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:51.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:51.620 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:51.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:51.620 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:51.310+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:52.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:51.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:52.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:51.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:52.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:51.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:52.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:51.599+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:52.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:52.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:52.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:52.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:52.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:52.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:52.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:52.008+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:52.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:52.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:52.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:52.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:52.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:52.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:52.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:52.322+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:53.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:52.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:53.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:52.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:53.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:52.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:53.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:52.619+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:53.309 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:52.984+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2509 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:53.309 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:52.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:53.309 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:52.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:53.309 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:52.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:53.310 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:52.982+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:53.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:53.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:53.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:53.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:53.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:53.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:53.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:53.288+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:53.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:53.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:53.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:53.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:53.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:53.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:53.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:53.610+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:54.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:53.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:54.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:53.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:54.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:53.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:54.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:53.966+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:54.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:54.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:54.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:54.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:54.261+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:54.976 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:54.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:54.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:54.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:54.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:54.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:54.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:54.600+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:55.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:54.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:55.240 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:54.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:55.240 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:54.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:55.240 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:54.955+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:55.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:55.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:55.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:55.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:55.218+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:55.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:55.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:55.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:55.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:55.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:55.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:55.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:55.603+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:56.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:55.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:56.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:55.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:56.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:55.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:56.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:55.913+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:56.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:56.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:56.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:56.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:56.265+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:56.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:56.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:56.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:56.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:56.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:56.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:56.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:56.614+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:57.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:56.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:57.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:56.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:57.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:56.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:57.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:56.905+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:57.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:57.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:57.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:57.244+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:57.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:57.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:57.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:57.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:57.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:57.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:57.883 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:57.577+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:58.133 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:09:57.985+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2602 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:09:58.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:57.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:58.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:57.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:58.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:57.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:58.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:57.862+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:58.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:58.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:58.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:58.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:58.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:58.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:58.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:58.245+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:58.909 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:58.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:58.909 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:58.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:58.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:58.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:58.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:58.574+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:59.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:58.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:59.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:58.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:59.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:58.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:09:59.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:58.888+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:09:59.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:59.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:59.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:59.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:59.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:59.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:09:59.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:09:59.205+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:09:59.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:59.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:59.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:59.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:59.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:59.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:09:59.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:09:59.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:00.273 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:59.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:00.273 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:59.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:00.273 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:59.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:00.273 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:09:59.868+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:00.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:00.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:00.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:00.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:00.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:00.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:00.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:00.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:00.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:00.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:00.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:00.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:00.575+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:01.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:00.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:01.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:00.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:01.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:00.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:01.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:00.871+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:01.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:01.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:01.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:01.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:01.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:01.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:01.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:01.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:01.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:01.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:01.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:01.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:01.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:01.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:01.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:01.598+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:02.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:01.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:02.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:01.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:02.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:01.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:02.311 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:01.897+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:02.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:02.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:02.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:02.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:02.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:02.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:02.290+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:02.954 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:02.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:02.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:02.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:02.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:02.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:02.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:02.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:03.263 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:02.986+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2696 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:03.264 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:02.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:03.264 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:02.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:03.264 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:02.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:03.264 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:02.933+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:03.559 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:03.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:03.559 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:03.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:03.559 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:03.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:03.559 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:03.242+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:03.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:03.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:03.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:03.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:03.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:03.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:03.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:03.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:04.223 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:03.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:04.223 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:03.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:04.223 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:03.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:04.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:03.934+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:04.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:04.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:04.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:04.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:04.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:04.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:04.202+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:04.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:04.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:04.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:04.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:04.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:04.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:04.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:04.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:05.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:04.922+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:05.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:04.922+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:05.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:04.922+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:05.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:04.923+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:05.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:05.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:05.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:05.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:05.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:05.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:05.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:05.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:05.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:05.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:05.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:05.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:05.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:05.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:05.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:05.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:06.220 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:05.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:06.220 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:05.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:06.220 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:05.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:06.220 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:05.912+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:06.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:06.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:06.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:06.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:06.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:06.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:06.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:06.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:06.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:06.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:06.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:06.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:06.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:06.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:06.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:06.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:07.197 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:06.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:07.197 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:06.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:07.197 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:06.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:07.197 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:06.911+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:07.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:07.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:07.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:07.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:07.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:07.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:07.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:07.176+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:07.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:07.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:07.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:07.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:07.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:07.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:07.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:07.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:08.312 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:07.987+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2790 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:08.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:07.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:08.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:07.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:08.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:07.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:08.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:07.939+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:08.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:08.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:08.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:08.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:08.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:08.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:08.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:08.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:08.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:08.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:08.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:08.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:08.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:08.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:08.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:08.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:09.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:08.980+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:09.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:08.980+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:09.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:08.980+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:09.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:08.980+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:09.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:09.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:09.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:09.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:09.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:09.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:09.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:09.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:09.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:09.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:09.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:09.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:09.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:09.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:09.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:09.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:10.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:09.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:10.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:09.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:10.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:09.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:10.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:09.975+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:10.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:10.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:10.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:10.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:10.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:10.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:10.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:10.170+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:10.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:10.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:10.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:10.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:10.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:10.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:10.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:10.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:11.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:10.939+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:11.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:10.939+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:11.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:10.939+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:11.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:10.940+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:11.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:11.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:11.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:11.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:11.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:11.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:11.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:11.138+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:11.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:11.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:11.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:11.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:11.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:11.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:11.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:11.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:12.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:11.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:12.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:11.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:12.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:11.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:12.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:11.917+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:12.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:12.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:12.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:12.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:12.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:12.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:12.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:12.172+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:12.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:12.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:12.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:12.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:12.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:12.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:12.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:12.455+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:13.211 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:12.988+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2885 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:13.211 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:12.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:13.211 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:12.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:13.211 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:12.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:13.211 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:12.892+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:13.481 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:13.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:13.481 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:13.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:13.481 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:13.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:13.481 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:13.190+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:13.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:13.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:13.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:13.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:13.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:13.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:13.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:13.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:14.248 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:13.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:14.248 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:13.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:14.248 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:13.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:14.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:13.858+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:14.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:14.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:14.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:14.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:14.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:14.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:14.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:14.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:14.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:14.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:14.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:14.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:15.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:14.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:15.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:14.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:15.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:14.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:15.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:14.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:15.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:15.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:15.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:15.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:15.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:15.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:15.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:15.273+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:15.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:15.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:15.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:15.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:15.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:15.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:15.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:15.462+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:16.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:15.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:16.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:15.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:16.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:15.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:16.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:15.839+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:16.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:16.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:16.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:16.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:16.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:16.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:16.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:16.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:16.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:16.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:16.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:16.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:16.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:16.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:16.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:16.496+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:17.281 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:16.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:17.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:16.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:17.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:16.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:17.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:16.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:17.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:17.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:17.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:17.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:17.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:17.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:17.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:17.260+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:17.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:17.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:17.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:17.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:17.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:17.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:17.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:17.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:18.025 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:17.989+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 2978 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:18.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:17.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:18.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:17.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:18.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:17.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:18.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:17.753+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:18.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:18.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:18.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:18.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:18.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:18.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:18.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:18.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:18.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:18.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:18.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:18.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:18.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:18.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:18.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:18.504+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:19.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:18.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:19.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:18.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:19.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:18.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:19.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:18.782+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:19.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:19.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:19.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:19.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:19.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:19.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:19.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:19.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:19.768 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:19.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:19.768 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:19.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:19.768 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:19.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:19.768 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:19.295+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:20.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:19.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:20.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:19.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:20.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:19.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:20.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:19.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:20.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:20.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:20.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:20.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:20.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:20.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:20.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:20.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:21.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:20.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:21.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:20.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:21.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:20.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:21.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:20.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:21.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:20.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:21.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:20.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:21.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:20.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:21.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:20.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:21.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:21.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:21.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:21.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:21.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:21.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:21.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:21.306+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:22.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:21.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:22.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:21.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:22.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:21.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:22.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:21.803+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:22.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:21.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:22.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:21.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:22.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:21.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:22.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:21.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:22.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:22.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:22.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:22.352+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:22.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:22.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:22.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:22.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:23.137 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:22.990+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3073 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:23.137 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:22.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:23.138 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:22.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:23.138 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:22.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:23.138 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:22.838+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:23.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:23.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:23.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:23.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:23.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:23.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:23.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:23.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:23.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:23.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:23.385+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:24.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:23.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:24.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:23.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:24.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:23.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:24.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:23.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:24.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:24.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:24.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:24.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:24.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:24.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:24.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:24.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:24.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:24.354+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:24.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:24.354+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:24.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:24.354+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:24.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:24.355+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:25.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:24.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:25.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:24.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:25.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:24.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:25.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:24.838+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:25.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:25.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:25.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:25.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:25.512+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:25.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:25.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:25.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:25.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:25.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:25.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:25.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:25.315+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:26.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:25.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:26.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:25.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:26.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:25.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:26.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:25.872+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:26.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:26.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:26.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:26.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:26.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:26.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:26.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:26.344+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:26.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:26.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:26.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:26.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:26.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:26.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:26.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:26.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:27.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:26.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:27.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:26.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:27.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:26.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:27.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:26.907+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:27.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:27.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:27.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:27.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:27.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:27.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:27.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:27.339+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:27.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:27.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:27.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:27.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:27.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:27.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:27.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:27.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:28.312 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:27.991+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3166 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:28.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:27.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:28.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:27.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:28.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:27.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:28.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:27.872+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:28.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:28.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:28.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:28.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:28.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:28.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:28.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:28.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:28.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:28.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:28.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:28.549+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:29.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:28.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:29.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:28.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:29.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:28.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:29.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:28.915+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:29.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:29.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:29.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:29.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:29.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:29.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:29.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:29.350+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:29.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:29.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:29.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:29.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:30.246 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:29.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:30.247 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:29.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:30.247 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:29.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:30.247 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:29.902+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:30.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:30.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:30.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:30.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:30.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:30.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:30.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:30.313+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:30.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:30.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:30.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:30.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:30.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:30.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:30.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:30.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:31.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:30.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:31.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:30.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:31.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:30.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:31.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:30.947+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:31.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:31.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:31.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:31.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:31.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:31.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:31.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:31.338+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:31.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:31.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:31.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:31.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:31.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:31.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:31.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:31.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:32.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:31.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:32.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:31.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:32.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:31.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:32.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:31.946+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:32.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:32.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:32.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:32.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:32.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:32.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:32.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:32.337+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:32.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:32.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:32.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:32.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:32.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:32.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:32.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:32.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:33.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:32.992+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3261 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:33.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:32.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:33.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:32.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:33.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:32.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:33.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:32.927+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:33.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:33.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:33.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:33.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:33.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:33.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:33.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:33.319+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:33.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:33.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:33.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:33.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:33.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:33.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:33.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:33.421+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:34.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:33.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:34.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:33.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:34.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:33.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:34.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:33.906+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:34.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:34.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:34.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:34.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:34.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:34.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:34.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:34.434+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:34.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:34.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:34.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:34.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:34.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:34.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:34.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:34.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:35.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:34.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:35.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:34.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:35.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:34.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:35.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:34.861+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:35.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:35.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:35.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:35.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:35.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:35.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:35.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:35.453+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:35.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:35.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:35.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:35.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:35.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:35.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:35.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:35.373+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:36.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:35.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:36.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:35.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:36.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:35.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:36.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:35.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:36.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:36.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:36.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:36.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:36.495+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:36.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:36.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:36.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:36.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:36.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:36.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:36.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:36.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:37.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:36.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:37.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:36.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:37.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:36.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:37.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:36.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:37.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:37.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:37.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:37.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:37.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:37.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:37.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:37.535+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:37.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:37.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:37.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:37.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:37.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:37.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:37.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:37.371+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:38.140 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:37.993+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3354 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:38.140 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:37.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:38.140 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:37.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:38.140 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:37.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:38.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:37.858+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:38.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:38.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:38.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:38.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:38.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:38.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:38.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:38.322+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:38.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:38.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:38.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:38.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:38.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:38.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:38.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:38.496+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:39.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:38.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:39.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:38.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:39.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:38.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:39.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:38.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:39.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:39.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:39.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:39.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:39.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:39.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:39.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:39.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:39.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:39.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:39.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:39.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:39.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:39.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:39.347+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:40.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:39.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:40.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:39.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:40.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:39.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:40.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:39.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:40.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:40.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:40.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:40.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:40.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:40.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:40.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:40.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:40.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:40.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:40.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:40.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:40.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:40.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:40.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:40.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:41.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:40.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:41.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:40.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:41.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:40.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:41.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:40.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:41.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:41.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:41.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:41.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:41.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:41.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:41.733 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:41.409+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:41.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:41.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:41.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:41.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:41.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:41.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:41.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:41.560+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:42.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:41.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:42.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:41.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:42.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:41.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:42.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:41.711+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:42.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:42.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:42.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:42.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:42.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:42.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:42.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:42.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:42.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:42.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:42.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:42.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:43.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:42.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:43.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:42.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:43.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:42.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:43.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:42.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:43.312 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:42.994+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3449 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:43.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:43.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:43.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:43.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:43.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:43.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:43.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:43.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:43.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:43.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:43.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:43.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:43.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:43.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:43.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:43.494+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:44.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:43.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:44.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:43.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:44.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:43.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:44.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:43.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:44.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:44.311+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:44.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:44.311+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:44.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:44.311+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:44.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:44.312+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:44.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:44.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:44.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:44.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:44.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:44.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:44.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:44.540+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:45.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:44.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:45.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:44.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:45.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:44.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:45.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:44.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:45.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:45.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:45.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:45.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:45.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:45.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:45.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:45.283+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:46.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:45.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:46.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:45.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:46.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:45.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:46.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:45.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:45.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:45.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:45.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:45.562+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:46.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:46.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:46.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:46.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:46.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:46.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:46.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:46.285+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:47.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:46.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:47.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:46.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:47.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:46.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:47.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:46.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:47.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:46.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:47.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:46.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:47.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:46.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:47.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:46.566+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:47.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:47.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:47.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:47.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:47.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:47.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:47.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:47.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:47.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:47.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:47.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:47.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:47.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:47.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:47.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:47.550+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:48.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:47.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:48.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:47.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:48.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:47.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:48.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:47.646+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:48.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:47.996+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3542 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:48.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:48.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:48.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:48.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:48.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:48.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:48.287+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:48.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:48.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:48.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:48.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:48.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:48.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:48.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:48.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:48.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:48.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:48.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:48.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:49.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:49.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:49.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:49.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:49.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:49.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:49.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:49.246+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:49.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:49.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:49.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:49.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:49.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:49.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:49.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:49.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:49.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:49.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:49.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:49.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:50.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:50.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:50.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:50.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:50.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:50.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:50.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:50.231+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:50.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:50.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:50.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:50.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:50.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:50.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:50.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:50.489+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:50.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:50.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:50.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:50.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:50.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:50.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:50.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:50.652+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:51.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:51.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:51.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:51.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:51.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:51.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:51.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:51.203+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:51.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:51.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:51.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:51.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:51.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:51.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:51.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:51.516+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:51.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:51.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:51.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:51.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:51.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:51.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:51.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:51.648+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:52.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:52.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:52.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:52.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:52.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:52.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:52.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:52.217+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:52.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:52.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:52.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:52.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:52.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:52.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:52.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:52.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:52.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:52.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:52.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:52.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:52.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:52.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:52.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:52.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:53.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:52.997+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3637 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:53.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:53.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:53.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:53.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:53.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:53.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:53.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:53.443+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:53.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:53.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:53.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:53.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:53.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:53.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:53.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:53.255+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:54.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:53.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:54.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:53.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:54.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:53.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:54.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:53.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:54.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:54.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:54.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:54.234+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:54.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:54.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:54.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:54.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:54.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:54.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:54.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:54.468+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:55.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:54.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:55.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:54.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:55.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:54.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:55.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:54.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:55.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:55.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:55.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:55.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:55.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:55.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:55.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:55.282+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:55.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:55.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:55.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:55.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:55.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:55.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:55.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:55.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:56.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:55.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:56.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:55.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:56.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:55.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:56.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:55.697+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:56.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:56.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:56.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:56.235+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:56.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:56.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:56.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:56.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:56.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:56.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:56.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:56.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:57.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:56.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:57.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:56.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:57.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:56.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:57.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:56.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:57.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:57.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:57.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:57.262+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:57.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:57.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:57.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:57.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:57.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:57.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:57.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:57.477+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:58.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:57.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:58.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:57.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:58.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:57.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:58.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:57.662+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:58.301 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:10:57.998+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3731 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:10:58.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:58.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:58.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:58.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:58.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:58.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:58.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:58.280+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:10:58.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:58.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:58.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:58.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:58.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:58.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:58.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:58.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:59.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:58.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:59.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:58.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:59.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:58.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:10:59.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:58.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:59.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:59.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:59.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:59.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:59.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:59.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:10:59.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:10:59.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:10:59.637 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:59.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:59.637 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:59.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:59.637 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:59.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:10:59.637 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:10:59.320+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:00.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:59.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:00.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:59.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:00.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:59.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:00.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:10:59.615+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:00.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:00.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:00.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:00.282+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:00.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:00.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:00.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:00.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:00.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:00.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:00.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:00.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:01.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:00.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:01.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:00.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:01.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:00.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:01.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:00.584+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:01.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:01.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:01.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:01.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:01.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:01.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:01.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:01.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:01.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:01.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:01.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:01.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:01.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:01.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:01.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:01.316+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:02.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:01.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:02.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:01.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:02.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:01.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:02.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:01.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:02.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:02.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:02.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:02.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:02.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:02.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:02.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:02.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:02.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:02.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:02.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:02.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:02.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:02.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:02.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:02.475+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:03.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:02.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:03.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:02.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:03.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:02.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:03.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:02.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:03.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:02.999+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3829 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:03.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:03.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:03.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:03.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:03.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:03.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:03.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:03.489+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:03.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:03.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:03.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:03.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:03.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:03.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:03.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:03.384+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:04.038 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:03.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:04.038 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:03.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:04.038 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:03.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:04.038 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:03.681+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:04.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:04.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:04.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:04.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:04.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:04.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:04.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:04.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:04.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:04.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:04.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:04.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:04.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:04.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:04.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:04.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:05.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:04.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:05.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:04.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:05.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:04.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:05.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:04.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:05.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:05.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:05.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:05.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:05.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:05.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:05.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:05.404+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:05.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:05.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:05.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:05.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:05.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:05.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:05.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:05.361+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:06.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:05.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:06.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:05.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:06.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:05.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:06.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:05.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:06.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:06.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:06.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:06.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:06.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:06.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:06.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:06.409+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:06.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:06.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:06.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:06.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:06.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:06.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:06.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:06.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:07.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:06.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:07.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:06.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:07.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:06.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:07.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:06.770+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:07.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:07.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:07.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:07.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:07.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:07.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:07.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:07.457+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:07.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:07.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:07.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:07.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:07.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:07.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:07.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:07.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:08.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:07.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:08.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:07.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:08.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:07.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:08.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:07.726+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:08.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:08.000+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 3923 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:08.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:08.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:08.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:08.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:08.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:08.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:08.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:08.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:08.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:08.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:08.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:08.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:08.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:08.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:08.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:08.331+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:09.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:08.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:09.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:08.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:09.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:08.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:09.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:08.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:09.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:09.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:09.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:09.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:09.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:09.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:09.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:09.302+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:09.787 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:09.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:09.787 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:09.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:09.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:09.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:09.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:09.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:10.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:09.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:10.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:09.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:10.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:09.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:10.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:09.766+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:10.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:10.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:10.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:10.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:10.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:10.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:10.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:10.283+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:10.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:10.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:10.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:10.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:10.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:10.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:10.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:10.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:11.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:10.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:11.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:10.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:11.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:10.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:11.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:10.766+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:11.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:11.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:11.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:11.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:11.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:11.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:11.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:11.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:11.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:11.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:11.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:11.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:11.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:11.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:11.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:11.329+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:12.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:11.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:12.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:11.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:12.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:11.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:12.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:11.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:12.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:12.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:12.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:12.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:12.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:12.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:12.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:12.400+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:12.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:12.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:12.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:12.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:12.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:12.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:12.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:12.316+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:13.046 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:13.001+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4024 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:12.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:12.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:12.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:12.764+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:13.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:13.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:13.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:13.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:13.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:13.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:13.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:13.303+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:13.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:13.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:13.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:13.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:13.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:13.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:13.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:13.446+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:14.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:13.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:14.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:13.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:14.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:13.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:14.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:13.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:14.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:14.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:14.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:14.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:14.286+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:14.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:14.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:14.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:14.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:14.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:14.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:14.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:14.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:15.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:14.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:15.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:14.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:15.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:14.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:15.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:14.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:15.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:15.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:15.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:15.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:15.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:15.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:15.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:15.287+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:15.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:15.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:15.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:15.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:15.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:15.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:15.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:15.485+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:16.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:15.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:16.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:15.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:16.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:15.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:16.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:15.820+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:16.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:16.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:16.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:16.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:16.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:16.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:16.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:16.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:16.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:16.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:16.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:16.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:16.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:16.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:16.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:16.281+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:17.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:16.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:17.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:16.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:17.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:16.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:17.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:16.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:17.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:17.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:17.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:17.419+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:17.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:17.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:17.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:17.324+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:18.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:18.003+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4133 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:18.151 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:17.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:18.151 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:17.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:18.151 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:17.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:18.151 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:17.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:18.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:18.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:18.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:18.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:18.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:18.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:18.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:18.280+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:18.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:18.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:18.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:18.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:18.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:18.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:18.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:18.406+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:19.272 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:18.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:19.272 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:18.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:19.272 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:18.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:19.273 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:18.838+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:19.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:19.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:19.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:19.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:19.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:19.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:19.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:19.251+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:19.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:19.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:19.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:19.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:19.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:19.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:19.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:19.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:20.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:19.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:20.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:19.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:20.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:19.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:20.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:19.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:20.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:20.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:20.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:20.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:20.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:20.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:20.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:20.408+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:20.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:20.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:20.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:20.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:20.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:20.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:20.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:20.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:21.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:20.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:21.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:20.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:21.182 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:20.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:21.182 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:20.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:21.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:21.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:21.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:21.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:21.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:21.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:21.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:21.160+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:21.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:21.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:21.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:21.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:21.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:21.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:21.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:21.367+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:22.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:21.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:22.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:21.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:22.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:21.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:22.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:21.847+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:22.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:22.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:22.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:22.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:22.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:22.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:22.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:22.190+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:22.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:22.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:22.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:22.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:22.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:22.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:22.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:22.364+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:23.150 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:23.004+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4227 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:23.151 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:22.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:23.151 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:22.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:23.151 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:22.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:23.151 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:22.855+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:23.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:23.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:23.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:23.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:23.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:23.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:23.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:23.174+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:23.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:23.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:23.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:23.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:23.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:23.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:23.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:23.331+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:24.200 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:23.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:24.200 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:23.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:24.201 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:23.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:24.201 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:23.851+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:24.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:24.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:24.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:24.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:24.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:24.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:24.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:24.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:24.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:24.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:24.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:24.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:24.285+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:25.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:24.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:25.223 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:24.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:25.223 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:24.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:25.223 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:24.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:25.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:25.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:25.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:25.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:25.330+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:25.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:25.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:25.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:25.201+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:26.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:25.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:26.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:25.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:26.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:25.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:26.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:25.805+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:26.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:26.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:26.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:26.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:26.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:26.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:26.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:26.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:26.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:26.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:26.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:26.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:26.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:26.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:26.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:26.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:27.174 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:26.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:27.174 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:26.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:27.174 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:26.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:27.174 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:26.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:27.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:27.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:27.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:27.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:27.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:27.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:27.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:27.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:27.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:27.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:27.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:27.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:27.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:27.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:27.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:27.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:28.129 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:28.005+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4321 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:28.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:27.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:28.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:27.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:28.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:27.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:28.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:27.842+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:28.405 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:28.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:28.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:28.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:28.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:28.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:28.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:28.107+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:28.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:28.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:28.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:28.384+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:29.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:28.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:29.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:28.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:29.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:28.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:29.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:28.875+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:29.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:29.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:29.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:29.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:29.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:29.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:29.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:29.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:29.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:29.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:29.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:29.393+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:30.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:29.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:30.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:29.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:30.135 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:29.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:30.135 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:29.857+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:30.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:30.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:30.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:30.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:30.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:30.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:30.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:30.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:30.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:30.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:30.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:30.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:30.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:30.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:30.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:30.404+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:31.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:30.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:31.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:30.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:31.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:30.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:31.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:30.820+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:31.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:31.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:31.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:31.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:31.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:31.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:31.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:31.158+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:31.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:31.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:31.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:31.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:31.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:31.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:31.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:31.431+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:32.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:31.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:32.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:31.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:32.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:31.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:32.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:31.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:32.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:32.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:32.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:32.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:32.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:32.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:32.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:32.147+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:32.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:32.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:32.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:32.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:32.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:32.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:32.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:32.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:33.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:33.006+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4415 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:33.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:32.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:33.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:32.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:33.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:32.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:33.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:32.764+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:33.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:33.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:33.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:33.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:33.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:33.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:33.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:33.157+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:33.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:33.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:33.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:33.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:33.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:33.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:33.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:33.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:34.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:33.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:34.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:33.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:34.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:33.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:34.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:33.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:34.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:34.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:34.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:34.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:34.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:34.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:34.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:34.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:34.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:34.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:34.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:34.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:34.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:34.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:34.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:34.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:35.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:34.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:35.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:34.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:35.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:34.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:35.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:34.729+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:35.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:35.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:35.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:35.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:35.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:35.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:35.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:35.100+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:35.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:35.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:35.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:35.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:35.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:35.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:35.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:35.384+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:36.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:35.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:36.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:35.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:36.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:35.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:36.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:35.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:36.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:36.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:36.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:36.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:36.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:36.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:36.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:36.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:36.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:36.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:36.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:36.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:36.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:36.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:36.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:36.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:36.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:36.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:36.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:36.758+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:37.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:37.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:37.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:37.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:37.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:37.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:37.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:37.025+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:37.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:37.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:37.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:37.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:37.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:37.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:37.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:37.386+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:38.063 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:38.007+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4528 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:38.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:37.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:38.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:37.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:38.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:37.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:38.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:37.789+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:38.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:38.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:38.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:38.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:38.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:38.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:38.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:38.076+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:38.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:38.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:38.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:38.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:38.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:38.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:38.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:38.354+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:39.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:38.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:39.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:38.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:39.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:38.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:39.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:38.780+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:39.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:39.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:39.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:39.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:39.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:39.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:39.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:39.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:39.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:39.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:39.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:39.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:39.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:39.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:39.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:39.397+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:40.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:39.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:40.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:39.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:40.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:39.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:40.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:39.780+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:40.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:40.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:40.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:40.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:40.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:40.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:40.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:40.085+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:40.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:40.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:40.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:40.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:40.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:40.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:40.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:40.354+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:41.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:40.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:41.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:40.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:41.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:40.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:41.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:40.820+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:41.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:41.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:41.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:41.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:41.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:41.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:41.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:41.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:41.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:41.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:41.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:41.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:41.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:41.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:41.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:41.389+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:42.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:41.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:42.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:41.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:42.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:41.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:42.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:41.802+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:42.387 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:42.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:42.387 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:42.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:42.387 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:42.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:42.387 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:42.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:42.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:42.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:42.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:42.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:42.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:42.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:42.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:42.365+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:43.117 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:43.008+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4632 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:43.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:42.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:43.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:42.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:43.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:42.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:43.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:42.807+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:43.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:43.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:43.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:43.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:43.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:43.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:43.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:43.095+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:43.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:43.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:43.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:43.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:43.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:43.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:43.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:43.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:44.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:43.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:44.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:43.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:44.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:43.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:44.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:43.837+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:44.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:44.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:44.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:44.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:44.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:44.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:44.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:44.073+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:44.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:44.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:44.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:44.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:44.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:44.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:44.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:44.387+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:45.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:44.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:45.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:44.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:45.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:44.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:45.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:44.875+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:45.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:45.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:45.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:45.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:45.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:45.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:45.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:45.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:45.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:45.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:45.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:45.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:45.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:45.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:45.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:45.425+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:46.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:45.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:46.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:45.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:46.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:45.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:46.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:45.834+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:46.458 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:46.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:46.458 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:46.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:46.458 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:46.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:46.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:46.159+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:46.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:46.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:46.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:46.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:46.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:46.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:46.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:46.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:47.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:46.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:47.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:46.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:47.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:46.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:47.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:46.824+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:47.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:47.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:47.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:47.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:47.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:47.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:47.200+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:47.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:47.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:47.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:47.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:47.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:47.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:47.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:47.403+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:48.159 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:48.010+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4726 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:48.159 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:47.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:48.159 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:47.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:48.159 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:47.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:48.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:47.868+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:48.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:48.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:48.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:48.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:48.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:48.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:48.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:48.216+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:48.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:48.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:48.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:48.443+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:49.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:48.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:49.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:48.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:49.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:48.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:49.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:48.859+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:49.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:49.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:49.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:49.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:49.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:49.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:49.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:49.213+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:49.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:49.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:49.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:49.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:50.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:49.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:50.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:49.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:50.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:49.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:50.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:49.903+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:50.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:50.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:50.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:50.230+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:50.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:50.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:50.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:50.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:50.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:50.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:50.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:50.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:51.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:50.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:51.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:50.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:51.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:50.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:51.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:50.915+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:51.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:51.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:51.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:51.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:51.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:51.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:51.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:51.204+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:51.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:51.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:51.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:51.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:51.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:51.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:51.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:51.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:52.244 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:51.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:52.245 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:51.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:52.245 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:51.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:52.245 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:51.944+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:52.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:52.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:52.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:52.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:52.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:52.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:52.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:52.223+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:52.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:52.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:52.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:52.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:52.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:52.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:52.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:52.514+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:53.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:53.011+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4820 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:53.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:52.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:53.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:52.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:53.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:52.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:53.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:52.994+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:53.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:53.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:53.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:53.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:53.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:53.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:53.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:53.226+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:53.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:53.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:53.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:53.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:53.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:53.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:53.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:53.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:54.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:54.043+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:54.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:54.043+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:54.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:54.043+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:54.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:54.044+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:54.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:54.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:54.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:54.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:54.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:54.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:54.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:54.253+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:54.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:54.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:54.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:54.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:54.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:54.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:54.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:54.432+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:55.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:55.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:55.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:55.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:55.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:55.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:55.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:55.249+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:55.494 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:55.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:55.494 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:55.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:55.494 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:55.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:55.494 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:55.061+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:55.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:55.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:55.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:55.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:55.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:55.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:55.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:55.472+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:56.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:56.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:56.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:56.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:56.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:56.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:56.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:56.223+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:56.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:56.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:56.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:56.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:56.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:56.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:56.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:56.033+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:56.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:56.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:56.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:56.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:56.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:56.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:56.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:56.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:57.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:57.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:57.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:57.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:57.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:57.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:57.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:57.248+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:57.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:57.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:57.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:57.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:57.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:57.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:57.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:57.011+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:57.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:57.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:57.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:57.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:57.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:57.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:57.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:57.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:58.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:11:58.012+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 4914 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:11:58.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:58.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:58.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:58.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:58.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:58.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:58.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:58.213+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:58.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:57.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:58.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:57.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:58.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:57.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:58.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:57.975+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:58.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:58.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:58.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:58.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:59.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:58.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:59.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:58.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:59.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:58.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:11:59.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:58.952+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:11:59.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:59.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:59.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:59.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:59.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:59.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:11:59.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:11:59.230+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:11:59.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:59.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:59.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:59.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:59.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:59.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:11:59.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:11:59.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:59.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:59.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:59.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:11:59.959+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:00.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:00.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:00.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:00.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:00.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:00.993 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:00.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:00.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:00.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:00.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:00.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:00.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:00.545+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:01.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:00.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:01.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:00.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:01.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:00.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:01.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:00.972+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:01.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:01.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:01.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:01.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:01.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:01.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:01.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:01.269+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:01.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:01.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:01.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:01.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:01.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:01.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:01.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:01.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:02.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:01.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:02.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:01.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:02.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:01.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:02.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:01.993+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:02.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:02.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:02.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:02.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:02.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:02.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:02.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:02.261+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:02.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:02.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:02.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:02.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:03.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:03.013+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5008 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:03.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:02.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:03.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:02.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:03.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:02.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:03.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:02.994+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:03.567 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:03.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:03.567 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:03.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:03.568 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:03.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:03.568 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:03.504+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:03.568 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:03.310+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:03.568 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:03.310+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:03.568 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:03.310+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:03.568 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:03.310+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:04.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:03.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:04.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:03.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:04.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:03.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:04.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:03.961+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:04.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:04.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:04.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:04.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:04.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:04.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:04.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:04.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:04.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:04.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:04.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:04.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:04.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:04.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:04.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:04.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:05.306 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:04.922+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:05.307 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:04.922+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:05.307 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:04.922+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:05.307 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:04.923+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:05.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:05.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:05.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:05.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:05.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:05.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:05.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:05.285+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:05.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:05.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:05.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:05.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:05.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:05.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:05.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:05.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:06.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:05.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:06.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:05.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:06.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:05.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:06.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:05.960+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:06.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:06.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:06.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:06.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:06.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:06.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:06.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:06.316+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:06.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:06.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:06.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:06.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:06.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:06.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:06.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:06.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:07.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:06.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:07.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:06.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:07.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:06.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:07.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:06.928+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:07.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:07.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:07.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:07.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:07.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:07.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:07.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:07.307+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:07.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:07.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:07.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:07.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:07.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:07.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:07.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:07.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:08.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:08.015+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5102 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:08.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:07.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:08.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:07.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:08.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:07.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:08.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:07.970+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:08.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:08.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:08.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:08.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:08.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:08.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:08.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:08.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:08.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:08.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:08.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:08.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:08.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:08.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:08.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:08.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:09.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:08.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:09.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:08.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:09.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:08.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:09.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:08.977+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:09.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:09.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:09.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:09.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:09.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:09.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:09.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:09.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:09.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:09.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:09.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:09.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:09.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:09.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:09.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:09.367+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:10.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:09.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:10.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:09.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:10.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:09.936+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:10.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:09.937+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:10.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:10.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:10.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:10.392+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:10.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:10.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:10.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:10.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:10.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:10.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:10.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:10.415+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:11.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:10.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:11.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:10.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:11.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:10.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:11.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:10.966+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:11.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:11.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:11.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:11.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:11.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:11.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:11.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:11.386+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:11.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:11.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:11.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:11.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:11.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:11.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:11.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:11.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:12.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:12.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:12.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:12.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:12.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:12.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:12.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:12.011+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:12.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:12.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:12.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:12.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:12.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:12.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:12.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:12.340+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:12.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:12.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:12.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:12.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:12.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:12.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:12.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:12.435+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:13.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:13.016+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5196 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:13.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:12.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:13.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:12.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:13.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:12.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:13.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:12.994+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:13.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:13.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:13.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:13.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:13.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:13.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:13.381+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:13.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:13.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:13.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:13.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:14.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:14.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:14.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:14.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:14.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:14.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:14.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:14.029+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:14.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:14.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:14.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:14.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:14.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:14.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:14.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:14.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:14.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:14.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:14.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:14.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:14.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:14.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:14.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:14.417+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:15.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:15.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:15.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:15.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:15.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:15.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:15.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:15.020+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:15.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:15.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:15.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:15.433+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:15.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:15.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:15.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:15.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:16.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:16.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:16.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:16.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:16.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:16.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:16.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:16.002+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:16.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:16.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:16.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:16.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:16.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:16.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:16.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:16.397+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:16.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:16.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:16.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:16.435+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:17.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:16.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:17.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:16.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:17.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:16.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:17.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:16.991+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:17.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:17.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:17.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:17.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:17.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:17.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:17.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:17.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:17.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:17.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:17.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:17.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:17.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:17.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:17.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:18.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:18.017+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5290 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:18.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:17.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:18.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:17.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:18.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:17.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:18.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:17.992+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:18.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:18.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:18.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:18.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:18.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:18.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:18.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:18.434+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:19.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:19.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:19.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:19.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:19.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:19.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:19.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:19.035+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:19.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:19.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:19.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:19.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:19.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:19.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:19.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:19.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:19.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:19.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:19.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:19.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:19.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:19.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:19.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:19.440+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:20.365 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:20.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:20.365 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:20.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:20.365 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:20.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:20.365 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:20.058+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:20.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:20.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:20.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:20.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:20.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:20.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:20.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:20.343+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:20.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:20.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:20.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:20.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:20.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:20.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:20.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:20.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:21.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:21.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:21.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:21.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:21.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:21.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:21.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:21.078+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:21.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:21.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:21.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:21.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:21.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:21.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:21.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:21.382+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:21.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:21.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:21.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:21.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:21.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:21.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:21.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:21.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:22.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:22.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:22.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:22.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:22.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:22.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:22.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:22.073+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:22.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:22.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:22.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:22.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:22.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:22.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:22.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:22.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:22.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:22.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:22.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:22.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:23.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:23.019+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5384 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:23.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:23.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:23.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:23.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:23.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:23.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:23.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:23.053+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:23.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:23.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:23.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:23.408+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:23.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:23.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:23.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:23.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:23.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:23.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:23.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:23.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:24.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:24.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:24.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:24.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:24.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:24.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:24.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:24.076+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:24.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:24.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:24.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:24.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:24.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:24.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:24.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:24.395+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:24.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:24.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:24.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:24.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:24.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:24.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:24.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:24.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:25.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:25.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:25.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:25.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:25.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:25.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:25.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:25.061+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:25.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:25.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:25.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:25.527+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:25.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:25.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:25.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:25.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:25.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:25.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:25.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:25.414+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:26.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:26.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:26.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:26.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:26.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:26.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:26.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:26.031+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:26.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:26.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:26.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:26.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:26.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:26.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:26.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:26.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:26.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:26.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:26.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:26.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:26.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:26.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:26.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:26.401+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:27.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:27.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:27.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:27.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:27.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:27.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:27.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:27.065+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:27.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:27.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:27.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:27.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:27.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:27.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:27.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:27.376+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:27.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:27.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:27.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:27.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:27.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:27.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:27.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:27.538+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:28.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:28.020+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5478 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:28.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:28.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:28.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:28.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:28.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:28.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:28.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:28.052+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:28.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:28.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:28.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:28.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:28.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:28.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:28.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:28.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:28.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:28.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:28.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:28.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:28.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:28.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:28.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:28.538+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:29.373 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:29.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:29.373 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:29.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:29.373 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:29.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:29.373 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:29.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:29.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:29.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:29.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:29.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:29.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:29.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:29.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:29.351+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:29.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:29.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:29.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:29.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:29.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:29.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:29.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:29.574+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:30.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:30.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:30.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:30.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:30.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:30.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:30.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:30.144+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:30.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:30.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:30.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:30.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:30.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:30.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:30.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:30.302+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:31.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:30.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:31.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:30.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:31.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:30.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:31.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:30.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:31.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:31.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:31.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:31.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:31.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:31.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:31.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:31.128+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:31.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:31.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:31.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:31.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:31.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:31.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:31.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:31.294+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:31.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:31.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:31.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:32.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:31.605+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:32.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:32.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:32.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:32.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:32.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:32.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:32.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:32.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:32.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:32.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:32.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:32.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:32.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:32.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:32.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:32.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:32.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:32.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:32.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:32.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:33.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:33.021+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5572 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:33.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:33.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:33.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:33.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:33.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:33.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:33.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:33.168+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:33.569 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:33.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:33.569 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:33.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:33.569 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:33.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:33.569 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:33.343+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:34.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:33.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:34.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:33.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:34.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:33.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:34.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:33.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:34.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:34.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:34.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:34.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:34.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:34.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:34.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:34.139+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:34.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:34.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:34.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:34.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:34.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:34.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:34.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:34.353+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:35.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:34.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:35.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:34.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:35.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:34.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:35.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:34.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:35.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:35.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:35.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:35.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:35.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:35.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:35.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:35.187+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:35.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:35.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:35.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:35.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:35.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:35.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:35.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:35.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:36.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:35.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:36.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:35.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:36.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:35.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:36.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:35.670+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:36.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:36.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:36.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:36.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:36.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:36.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:36.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:36.177+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:36.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:36.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:36.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:36.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:36.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:36.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:36.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:36.372+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:36.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:37.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:36.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:37.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:36.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:37.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:36.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:37.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:37.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:37.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:37.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:37.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:37.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:37.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:37.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:37.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:37.182+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:37.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:37.182+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:37.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:37.182+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:37.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:37.183+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:38.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:37.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:38.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:37.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:38.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:37.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:38.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:37.684+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:38.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:38.023+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5666 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:38.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:38.221+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:38.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:38.222+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:38.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:38.222+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:38.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:38.223+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:38.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:38.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:38.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:38.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:38.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:38.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:38.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:38.333+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:39.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:38.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:39.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:38.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:39.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:38.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:39.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:38.670+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:39.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:39.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:39.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:39.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:39.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:39.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:39.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:39.286+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:39.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:39.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:39.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:39.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:39.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:39.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:39.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:39.187+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:39.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:39.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:39.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:39.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:39.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:39.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:39.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:39.681+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:40.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:40.220+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:40.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:40.220+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:40.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:40.220+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:40.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:40.221+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:40.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:40.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:40.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:40.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:40.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:40.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:40.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:40.280+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:40.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:40.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:40.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:41.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:40.670+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:41.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:41.178+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:41.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:41.178+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:41.178+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:41.179+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:41.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:41.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:41.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:41.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:41.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:41.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:41.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:41.263+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:41.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:41.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:41.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:41.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:42.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:42.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:42.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:42.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:42.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:42.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:42.165+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:42.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:42.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:42.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:42.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:42.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:42.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:42.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:42.284+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:42.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:42.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:42.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:42.656+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:43.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:43.024+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5760 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:43.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:43.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:43.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:43.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:43.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:43.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:43.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:43.118+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:43.653 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:43.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:43.654 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:43.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:43.654 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:43.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:43.654 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:43.331+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:44.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:43.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:44.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:43.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:44.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:43.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:44.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:43.631+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:44.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:44.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:44.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:44.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:44.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:44.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:44.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:44.161+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:44.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:44.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:44.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:44.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:44.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:44.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:44.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:44.290+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:45.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:44.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:45.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:44.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:45.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:44.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:45.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:44.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:45.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:45.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:45.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:45.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:45.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:45.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:45.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:45.123+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:45.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:45.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:45.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:45.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:45.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:45.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:45.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:45.324+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:45.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:45.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:45.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:45.663+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:46.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:46.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:46.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:46.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:46.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:46.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:46.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:46.148+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:46.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:46.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:46.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:46.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:46.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:46.306+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:46.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:46.307+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:47.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:46.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:47.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:46.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:47.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:46.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:47.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:46.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:47.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:47.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:47.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:47.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:47.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:47.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:47.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:47.198+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:47.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:47.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:47.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:47.298+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:48.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:47.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:48.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:47.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:48.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:47.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:48.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:47.711+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:48.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:48.025+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5854 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:48.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:48.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:48.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:48.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:48.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:48.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:48.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:48.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:48.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:48.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:48.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:48.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:48.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:48.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:48.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:48.182+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:48.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:48.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:48.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:48.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:48.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:49.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:49.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:49.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:49.202+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:49.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:49.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:49.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:49.253+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:49.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:49.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:49.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:49.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:49.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:49.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:49.661+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:50.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:50.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:50.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:50.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:50.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:50.278+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:50.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:50.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:50.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:50.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:50.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:50.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:50.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:50.175+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:51.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:50.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:51.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:50.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:51.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:50.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:51.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:50.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:51.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:51.191+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:51.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:51.191+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:51.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:51.191+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:51.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:51.192+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:51.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:51.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:51.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:51.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:51.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:51.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:51.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:51.325+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:52.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:51.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:52.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:51.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:52.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:51.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:52.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:51.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:52.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:52.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:52.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:52.167+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:52.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:52.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:52.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:52.304+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:53.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:52.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:53.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:52.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:53.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:52.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:53.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:52.655+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:53.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:53.027+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 5948 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:53.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:53.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:53.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:53.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:53.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:53.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:53.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:53.161+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:53.629 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:53.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:53.629 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:53.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:53.629 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:53.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:53.629 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:53.353+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:54.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:53.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:54.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:53.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:54.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:53.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:54.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:53.607+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:54.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:54.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:54.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:54.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:54.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:54.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:54.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:54.147+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:54.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:54.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:54.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:54.389+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:55.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:54.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:55.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:54.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:55.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:54.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:55.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:54.651+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:55.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:55.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:55.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:55.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:55.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:55.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:55.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:55.138+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:55.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:55.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:55.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:55.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:55.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:55.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:55.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:55.377+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:56.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:55.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:56.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:55.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:56.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:55.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:56.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:55.607+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:56.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:56.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:56.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:56.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:56.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:56.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:56.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:56.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:56.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:56.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:56.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:56.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:56.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:56.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:56.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:56.573+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:56.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:56.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:56.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:56.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:56.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:56.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:56.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:56.400+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:57.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:57.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:57.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:57.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:57.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:57.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:57.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:57.175+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:57.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:57.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:57.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:57.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:57.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:57.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:57.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:57.447+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:57.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:57.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:57.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:57.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:57.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:57.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:57.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:57.622+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:58.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:12:58.028+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6042 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:12:58.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:58.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:58.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:58.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:58.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:58.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:58.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:58.216+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:58.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:58.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:58.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:58.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:58.589+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:58.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:58.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:58.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:58.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:58.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:58.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:58.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:58.454+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:59.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:59.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:59.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:59.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:59.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:59.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:12:59.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:12:59.229+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:12:59.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:59.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:59.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:59.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:59.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:59.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:12:59.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:12:59.480+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:12:59.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:59.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:59.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:59.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:59.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:59.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:12:59.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:12:59.634+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:00.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:00.206+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:00.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:00.206+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:00.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:00.206+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:00.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:00.207+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:00.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:00.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:00.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:00.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:00.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:00.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:00.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:00.455+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:01.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:00.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:01.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:00.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:01.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:00.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:01.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:00.645+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:01.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:01.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:01.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:01.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:01.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:01.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:01.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:01.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:01.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:01.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:01.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:01.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:01.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:01.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:01.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:01.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:01.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:01.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:01.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:01.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:01.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:01.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:01.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:01.488+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:02.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:02.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:02.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:02.141+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:02.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:02.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:02.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:02.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:02.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:02.693+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:02.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:02.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:02.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:02.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:02.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:02.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:02.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:02.522+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:03.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:03.030+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6136 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:03.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:03.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:03.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:03.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:03.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:03.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:03.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:03.173+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:03.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:03.473+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:03.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:03.473+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:03.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:03.473+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:03.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:03.473+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:03.847 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:03.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:03.847 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:03.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:03.847 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:03.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:03.847 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:03.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:04.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:04.184+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:04.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:04.184+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:04.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:04.184+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:04.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:04.185+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:04.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:04.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:04.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:04.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:04.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:04.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:04.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:04.432+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:05.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:04.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:05.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:04.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:05.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:04.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:05.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:04.664+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:05.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:05.198+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:05.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:05.198+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:05.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:05.198+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:05.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:05.199+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:05.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:05.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:05.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:05.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:05.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:05.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:05.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:05.436+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:06.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:05.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:06.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:05.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:06.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:05.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:06.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:05.638+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:06.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:06.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:06.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:06.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:06.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:06.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:06.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:06.194+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:06.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:06.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:06.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:06.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:06.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:06.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:06.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:06.644+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:06.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:06.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:06.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:06.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:06.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:06.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:06.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:06.473+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:07.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:07.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:07.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:07.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:07.203+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:07.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:07.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:07.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:07.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:07.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:07.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:07.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:07.689+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:07.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:07.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:07.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:07.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:07.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:07.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:07.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:07.504+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:08.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:08.031+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6230 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:08.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:08.228+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:08.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:08.228+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:08.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:08.228+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:08.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:08.228+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:09.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:08.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:09.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:08.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:09.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:08.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:09.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:08.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:09.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:08.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:09.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:08.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:09.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:08.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:09.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:08.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:09.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:09.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:09.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:09.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:09.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:09.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:09.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:09.183+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:09.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:09.538+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:09.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:09.538+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:09.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:09.539+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:09.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:09.539+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:09.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:09.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:09.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:09.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:09.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:09.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:09.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:09.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:10.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:10.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:10.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:10.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:10.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:10.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:10.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:10.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:11.063 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:10.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:11.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:10.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:11.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:10.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:11.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:10.543+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:11.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:10.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:11.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:10.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:11.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:10.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:11.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:10.613+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:11.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:11.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:11.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:11.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:11.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:11.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:11.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:11.151+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:12.063 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:11.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:12.063 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:11.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:12.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:11.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:12.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:11.571+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:12.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:11.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:12.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:11.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:12.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:11.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:12.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:11.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:12.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:12.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:12.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:12.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:12.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:12.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:12.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:12.124+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:12.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:12.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:12.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:12.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:12.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:12.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:12.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:12.538+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:12.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:12.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:12.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:12.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:12.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:12.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:12.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:12.529+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:13.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:13.032+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6324 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:13.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:13.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:13.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:13.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:13.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:13.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:13.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:13.165+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:13.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:13.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:13.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:13.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:13.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:13.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:13.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:13.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:13.505+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:14.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:14.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:14.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:14.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:14.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:14.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:14.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:14.121+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:14.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:14.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:14.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:14.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:14.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:14.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:14.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:14.484+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:14.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:14.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:14.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:14.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:14.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:14.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:14.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:14.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:15.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:15.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:15.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:15.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:15.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:15.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:15.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:15.138+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:15.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:15.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:15.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:15.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:15.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:15.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:15.499+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:15.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:15.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:15.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:15.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:16.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:16.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:16.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:16.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:16.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:16.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:16.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:16.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:16.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:16.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:16.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:16.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:16.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:16.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:16.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:16.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:16.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:16.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:16.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:16.543+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:17.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:17.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:17.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:17.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:17.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:17.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:17.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:17.181+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:17.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:17.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:17.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:17.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:17.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:17.484+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:17.484+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:17.484+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:17.484+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:18.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:18.034+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6418 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:18.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:18.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:18.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:18.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:18.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:18.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:18.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:18.210+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:18.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:18.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:18.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:18.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:18.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:18.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:18.469+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:18.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:18.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:18.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:18.543+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:19.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:19.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:19.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:19.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:19.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:19.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:19.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:19.241+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:19.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:19.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:19.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:19.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:19.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:19.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:19.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:19.440+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:19.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:19.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:19.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:19.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:19.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:19.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:19.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:19.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:20.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:20.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:20.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:20.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:20.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:20.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:20.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:20.475+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:20.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:20.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:20.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:20.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:20.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:20.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:20.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:20.226+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:20.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:20.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:20.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:20.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:20.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:20.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:20.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:20.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:21.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:21.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:21.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:21.202+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:21.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:21.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:21.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:21.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:21.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:21.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:21.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:21.432+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:21.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:21.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:21.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:21.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:21.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:21.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:21.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:21.504+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:22.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:22.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:22.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:22.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:22.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:22.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:22.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:22.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:22.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:22.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:22.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:22.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:22.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:22.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:22.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:22.215+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:22.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:22.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:22.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:22.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:22.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:22.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:22.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:23.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:23.035+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6512 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:23.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:23.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:23.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:23.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:23.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:23.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:23.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:23.258+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:23.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:23.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:23.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:23.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:23.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:23.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:23.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:23.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:23.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:23.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:23.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:23.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:23.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:23.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:23.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:23.561+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:24.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:24.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:24.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:24.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:24.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:24.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:24.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:24.233+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:24.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:24.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:24.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:24.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:24.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:24.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:24.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:24.385+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:24.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:24.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:24.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:24.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:24.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:24.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:24.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:24.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:25.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:25.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:25.553 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:25.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:25.553 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:25.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:25.553 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:25.242+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:25.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:25.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:25.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:25.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:25.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:25.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:25.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:25.361+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:25.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:25.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:25.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:25.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:25.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:25.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:26.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:26.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:26.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:26.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:26.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:26.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:26.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:26.275+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:26.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:26.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:26.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:26.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:26.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:26.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:26.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:26.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:26.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:26.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:26.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:26.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:26.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:26.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:26.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:26.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:27.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:27.325+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:27.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:27.325+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:27.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:27.325+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:27.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:27.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:27.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:27.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:27.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:27.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:27.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:27.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:27.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:27.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:27.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:27.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:27.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:27.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:27.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:27.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:27.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:27.384+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:28.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:28.037+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6605 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:28.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:28.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:28.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:28.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:28.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:28.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:28.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:28.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:28.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:28.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:28.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:28.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:28.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:28.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:28.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:28.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:28.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:28.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:28.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:28.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:28.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:28.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:28.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:28.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:29.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:29.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:29.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:29.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:29.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:29.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:29.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:29.335+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:29.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:29.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:29.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:29.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:29.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:29.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:29.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:29.343+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:29.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:29.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:29.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:29.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:29.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:29.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:29.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:29.506+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:30.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:30.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:30.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:30.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:30.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:30.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:30.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:30.342+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:30.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:30.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:30.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:30.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:30.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:30.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:30.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:30.475+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:30.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:30.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:30.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:30.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:30.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:30.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:30.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:30.366+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:31.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:31.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:31.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:31.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:31.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:31.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:31.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:31.371+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:31.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:31.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:31.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:31.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:31.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:31.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:31.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:31.343+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:31.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:31.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:31.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:31.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:31.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:31.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:31.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:31.512+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:32.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:32.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:32.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:32.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:32.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:32.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:32.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:32.330+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:32.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:32.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:32.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:32.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:32.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:32.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:32.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:32.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:32.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:32.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:32.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:32.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:32.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:32.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:32.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:32.308+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:33.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:33.038+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6700 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:33.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:33.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:33.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:33.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:33.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:33.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:33.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:33.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:33.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:33.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:33.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:33.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:33.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:33.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:33.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:33.322+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:33.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:33.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:33.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:33.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:33.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:33.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:33.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:33.514+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:34.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:34.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:34.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:34.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:34.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:34.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:34.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:34.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:34.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:34.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:34.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:34.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:34.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:34.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:34.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:34.302+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:34.718 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:34.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:34.718 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:34.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:34.718 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:34.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:34.718 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:34.468+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:35.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:35.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:35.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:35.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:35.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:35.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:35.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:35.374+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:35.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:35.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:35.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:35.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:35.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:35.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:35.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:35.257+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:35.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:35.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:35.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:35.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:35.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:35.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:35.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:35.513+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:36.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:36.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:36.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:36.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:36.495+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:36.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:36.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:36.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:36.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:36.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:36.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:36.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:36.336+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:36.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:36.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:36.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:36.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:36.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:36.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:36.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:36.270+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:37.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:37.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:37.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:37.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:37.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:37.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:37.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:37.255+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:37.553 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:37.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:37.553 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:37.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:37.554 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:37.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:37.554 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:37.349+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:37.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:37.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:37.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:37.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:37.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:37.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:37.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:37.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:38.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:38.040+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6793 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:38.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:38.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:38.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:38.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:38.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:38.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:38.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:38.292+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:38.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:38.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:38.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:38.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:38.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:38.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:38.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:38.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:38.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:38.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:38.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:38.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:38.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:38.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:38.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:38.350+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:39.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:39.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:39.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:39.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:39.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:39.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:39.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:39.307+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:39.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:39.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:39.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:39.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:39.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:39.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:39.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:39.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:39.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:39.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:39.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:39.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:39.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:39.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:39.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:39.329+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:40.712 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:40.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:40.712 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:40.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:40.712 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:40.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:40.712 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:40.299+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:40.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:40.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:40.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:40.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:40.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:40.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:40.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:40.529+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:40.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:40.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:40.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:40.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:40.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:40.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:40.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:40.324+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:41.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:41.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:41.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:41.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:41.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:41.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:41.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:41.273+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:41.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:41.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:41.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:41.298+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:42.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:41.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:42.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:41.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:42.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:41.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:42.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:41.545+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:42.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:42.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:42.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:42.318+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:42.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:42.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:42.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:42.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:42.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:42.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:42.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:42.232+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:43.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:42.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:43.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:42.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:43.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:42.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:43.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:42.580+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:43.320 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:43.041+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6887 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:43.320 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:43.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:43.320 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:43.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:43.321 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:43.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:43.321 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:43.219+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:43.573 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:43.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:43.573 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:43.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:43.573 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:43.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:43.573 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:43.298+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:44.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:43.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:44.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:43.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:44.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:43.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:44.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:43.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:44.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:44.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:44.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:44.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:44.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:44.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:44.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:44.297+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:44.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:44.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:44.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:44.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:44.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:44.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:44.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:44.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:44.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:44.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:44.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:44.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:44.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:44.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:44.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:44.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:45.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:45.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:45.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:45.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:45.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:45.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:45.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:45.347+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:45.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:45.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:45.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:45.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:45.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:45.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:45.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:45.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:45.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:45.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:45.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:45.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:45.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:45.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:45.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:45.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:46.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:46.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:46.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:46.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:46.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:46.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:46.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:46.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:46.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:46.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:46.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:46.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:46.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:46.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:46.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:46.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:46.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:46.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:46.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:46.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:46.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:46.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:46.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:46.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:47.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:47.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:47.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:47.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:47.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:47.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:47.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:47.365+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:47.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:47.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:47.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:47.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:48.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:47.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:48.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:47.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:48.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:47.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:48.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:47.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:48.422 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:48.043+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 6981 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:48.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:48.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:48.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:48.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:48.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:48.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:48.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:48.194+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:48.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:48.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:48.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:48.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:48.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:48.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:48.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:48.400+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:48.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:48.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:48.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:48.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:48.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:48.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:48.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:48.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:49.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:49.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:49.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:49.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:49.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:49.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:49.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:49.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:49.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:49.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:49.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:49.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:49.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:49.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:49.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:49.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:49.591+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:50.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:50.196+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:50.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:50.196+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:50.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:50.196+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:50.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:50.197+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:50.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:50.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:50.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:50.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:50.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:50.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:50.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:50.400+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:50.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:50.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:50.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:50.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:50.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:50.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:50.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:50.614+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:51.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:51.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:51.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:51.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:51.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:51.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:51.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:51.393+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:51.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:51.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:51.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:51.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:51.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:51.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:51.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:51.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:52.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:51.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:52.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:51.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:52.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:51.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:52.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:51.583+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:52.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:52.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:52.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:52.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:52.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:52.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:52.364+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:52.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:52.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:52.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:52.223+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:53.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:52.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:53.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:52.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:53.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:52.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:53.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:52.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:53.374 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:53.044+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7076 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:53.374 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:53.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:53.374 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:53.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:53.374 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:53.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:53.374 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:53.260+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:53.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:53.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:53.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:53.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:53.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:53.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:53.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:53.352+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:53.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:53.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:53.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:53.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:53.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:53.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:53.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:53.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:54.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:54.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:54.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:54.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:54.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:54.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:54.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:54.330+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:54.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:54.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:54.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:54.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:55.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:54.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:55.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:54.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:55.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:54.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:55.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:54.542+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:55.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:55.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:55.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:55.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:55.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:55.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:55.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:55.308+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:55.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:55.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:55.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:55.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:55.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:55.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:55.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:55.309+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:55.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:55.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:55.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:55.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:55.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:55.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:55.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:55.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:56.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:56.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:56.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:56.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:56.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:56.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:56.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:56.293+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:56.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:56.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:56.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:56.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:56.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:56.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:56.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:56.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:56.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:56.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:56.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:56.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:56.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:56.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:56.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:56.501+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:57.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:57.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:57.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:57.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:57.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:57.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:57.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:57.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:57.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:57.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:57.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:57.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:57.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:57.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:57.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:57.282+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:57.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:57.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:57.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:57.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:57.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:57.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:57.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:57.472+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:58.351 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:13:58.045+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7169 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:13:58.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:58.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:58.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:58.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:58.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:58.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:58.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:58.329+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:58.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:58.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:58.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:58.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:58.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:58.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:58.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:58.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:58.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:58.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:58.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:58.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:58.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:58.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:58.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:58.453+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:59.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:59.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:59.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:59.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:59.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:59.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:13:59.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:13:59.307+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:13:59.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:59.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:59.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:59.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:59.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:59.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:13:59.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:13:59.419+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:13:59.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:59.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:59.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:59.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:59.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:59.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:13:59.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:13:59.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:00.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:00.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:00.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:00.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:00.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:00.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:00.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:00.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:00.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:00.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:00.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:00.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:00.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:00.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:00.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:00.374+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:00.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:00.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:00.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:00.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:00.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:00.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:00.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:00.464+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:01.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:01.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:01.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:01.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:01.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:01.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:01.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:01.389+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:01.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:01.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:01.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:01.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:01.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:01.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:01.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:01.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:01.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:01.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:01.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:01.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:01.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:01.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:01.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:01.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:02.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:02.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:02.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:02.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:02.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:02.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:02.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:02.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:02.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:02.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:02.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:02.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:02.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:02.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:02.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:02.358+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:02.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:02.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:02.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:02.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:02.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:02.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:02.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:02.366+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:03.363 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:03.047+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7263 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:03.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:03.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:03.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:03.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:03.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:03.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:03.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:03.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:03.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:03.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:03.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:03.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:03.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:03.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:03.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:03.550+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:03.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:03.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:03.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:03.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:03.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:03.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:03.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:03.342+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:04.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:04.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:04.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:04.309+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:04.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:04.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:04.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:04.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:04.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:04.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:04.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:04.402+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:04.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:04.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:04.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:04.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:04.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:04.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:04.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:04.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:05.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:05.433+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:05.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:05.433+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:05.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:05.433+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:05.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:05.433+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:05.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:05.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:05.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:05.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:05.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:05.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:05.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:05.492+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:05.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:05.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:05.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:05.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:05.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:05.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:05.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:05.334+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:06.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:06.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:06.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:06.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:06.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:06.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:06.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:06.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:06.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:06.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:06.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:06.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:06.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:06.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:06.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:06.288+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:06.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:06.437+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:06.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:06.437+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:06.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:06.437+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:06.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:06.438+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:07.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:07.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:07.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:07.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:07.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:07.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:07.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:07.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:07.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:07.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:07.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:07.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:07.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:07.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:07.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:07.275+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:07.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:07.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:07.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:07.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:07.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:07.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:07.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:07.450+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:08.466 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:08.048+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7357 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:08.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:08.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:08.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:08.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:08.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:08.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:08.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:08.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:08.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:08.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:08.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:08.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:08.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:08.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:08.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:08.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:08.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:08.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:08.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:08.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:08.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:08.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:08.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:08.494+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:09.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:09.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:09.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:09.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:09.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:09.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:09.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:09.301+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:09.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:09.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:09.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:09.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:09.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:09.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:09.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:09.501+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:09.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:09.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:09.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:09.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:09.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:09.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:09.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:09.494+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:10.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:10.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:10.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:10.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:10.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:10.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:10.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:10.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:10.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:10.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:10.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:10.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:10.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:10.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:10.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:10.538+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:10.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:10.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:10.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:10.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:10.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:10.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:10.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:10.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:11.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:11.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:11.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:11.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:11.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:11.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:11.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:11.313+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:11.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:11.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:11.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:11.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:11.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:11.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:11.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:11.521+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:11.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:11.517+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:11.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:11.517+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:11.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:11.517+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:11.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:11.518+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:12.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:12.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:12.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:12.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:12.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:12.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:12.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:12.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:12.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:12.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:12.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:12.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:12.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:12.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:12.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:12.496+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:12.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:12.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.074604+0000 front 2024-02-19T19:03:44.074541+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:12.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:12.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.074585+0000 front 2024-02-19T19:03:44.074526+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:12.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:12.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.074564+0000 front 2024-02-19T19:03:44.074619+0000 (oldest deadline 2024-02-19T19:04:08.174367+0000) 2024-02-19T19:14:12.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:12.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:13.331 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:13.050+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7452 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:13.527+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:13.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:13.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:13.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:13.482+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:13.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:13.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:13.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:13.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:13.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:13.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:13.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:13.309+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:14.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:14.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:14.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:14.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:14.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:14.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:14.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:14.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:14.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:14.302+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:14.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:14.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:14.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:14.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:14.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:14.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:14.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:14.441+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:15.496+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:15.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:15.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:15.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:15.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:15.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:15.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:15.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:15.432+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:15.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:15.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:15.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:15.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:15.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:15.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:15.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:15.334+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:16.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:16.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:16.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:44.284967+0000 front 2024-02-19T19:03:44.284993+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:16.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:16.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:44.285012+0000 front 2024-02-19T19:03:44.284935+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:16.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:16.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:44.285036+0000 front 2024-02-19T19:03:44.284952+0000 (oldest deadline 2024-02-19T19:04:10.184794+0000) 2024-02-19T19:14:16.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:16.323+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:16.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:16.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:16.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:16.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:16.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:16.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:16.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:16.432+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:17.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:17.279+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:17.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:17.423+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:17.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:17.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:17.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:17.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:17.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:17.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:17.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:17.457+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:18.332 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:18.052+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7544 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:18.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:18.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:18.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:18.388+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:18.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:18.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:18.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:18.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:18.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:18.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:18.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:18.461+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:19.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:19.341+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:19.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:19.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:19.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:19.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:19.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:19.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:19.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:19.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:19.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:19.417+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:20.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:20.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:20.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:20.401+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:20.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:20.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:20.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:20.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:20.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:20.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:20.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:20.411+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:21.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:21.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:21.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:21.285+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:21.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:21.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:21.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:21.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:21.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:21.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:21.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:21.382+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:22.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:22.270+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:22.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:22.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:22.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:22.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:22.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:22.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:22.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:22.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:22.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:22.401+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:23.384 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:23.053+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7639 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:23.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:23.259+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:23.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:23.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:23.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:23.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 since back 2024-02-19T19:03:47.041758+0000 front 2024-02-19T19:03:47.041863+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:23.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:23.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 since back 2024-02-19T19:03:47.041742+0000 front 2024-02-19T19:03:47.041775+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:23.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:23.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 since back 2024-02-19T19:03:47.041732+0000 front 2024-02-19T19:03:47.041785+0000 (oldest deadline 2024-02-19T19:04:11.141632+0000) 2024-02-19T19:14:23.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:23.425+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:24.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:24.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:24.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:24.322+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:24.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:24.385+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:25.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:25.344+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:25.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:25.373+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:25.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:25.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:26.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:26.368+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:26.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:26.365+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:26.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:26.366+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:27.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:27.324+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:27.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:27.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:27.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:27.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:28.340 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:28.055+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7733 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:28.340 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:28.301+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:28.347 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:14:28.322+0000 7f6ca550d700 0 monclient(hunting): authenticate timed out after 300 2024-02-19T19:14:28.347 INFO:teuthology.orchestra.run.smithi077.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-19T19:14:28.646 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:28.316+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:28.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:28.371+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:29.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-02-19T19:14:29.302 INFO:teuthology.orchestra.run.smithi077.stderr:Inferring config /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/mon.smithi077/config 2024-02-19T19:14:29.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:29.278+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:29.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:29.331+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:29.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:29.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:30.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:30.265+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:30.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:30.333+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:30.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:30.300+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:31.323 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:31.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:31.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:31.300+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:31.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:31.331+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:32.340 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:32.199+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:32.340 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:32.288+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:32.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:32.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:33.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:33.056+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7829 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:33.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:33.170+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:33.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:33.250+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:33.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:33.386+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:34.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:34.221+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:34.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:34.380+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:34.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:34.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:34.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:34.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:34.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:34.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:34.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:34.220+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:35.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:35.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:35.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:35.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:35.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:35.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:35.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:35.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:35.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:35.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:35.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:35.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:36.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:36.252+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:36.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:36.372+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:36.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:36.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:36.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:36.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:36.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:36.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:36.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:36.209+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:37.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:37.398+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:37.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:37.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:37.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:37.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:37.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:37.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:37.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:37.255+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:37.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:37.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:37.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:37.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:37.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:37.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:37.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:37.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:38.417 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:38.058+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 7921 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:38.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:38.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:38.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:38.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:38.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:38.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:38.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:38.219+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:38.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:38.278+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:38.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:38.278+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:38.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:38.278+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:38.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:38.278+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:38.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:38.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:39.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:39.433+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:39.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:39.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:39.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:39.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:39.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:39.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:39.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:39.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:39.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:39.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:39.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:39.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:39.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:39.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:39.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:39.198+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:40.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:40.405+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:40.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:40.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:40.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:40.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:40.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:40.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:40.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:40.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:40.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:40.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:40.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:40.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:40.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:40.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:40.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:40.349+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:41.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:41.401+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:41.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:41.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:41.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:41.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:41.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:41.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:41.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:41.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:41.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:41.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:41.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:41.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:41.362+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:42.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:42.426+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:42.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:42.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:42.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:42.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:42.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:42.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:42.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:42.274+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:42.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:42.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:42.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:42.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:42.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:43.343 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:43.059+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8016 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:43.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:43.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:43.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:43.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:43.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:43.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:43.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:43.238+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:43.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:43.430+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:43.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:43.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:43.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:43.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:43.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:43.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:43.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:43.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:44.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:44.395+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:44.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:44.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:44.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:44.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:44.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:44.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:44.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:44.211+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:44.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:44.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:44.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:44.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:44.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:44.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:44.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:44.385+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:45.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:45.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:45.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:45.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:45.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:45.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:45.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:45.218+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:45.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:45.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:45.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:45.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:45.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:45.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:45.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:45.409+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:45.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:45.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:45.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:45.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:45.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:45.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:45.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:45.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:46.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:46.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:46.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:46.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:46.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:46.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:46.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:46.248+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:46.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:46.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:46.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:46.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:46.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:46.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:46.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:46.410+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:46.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:46.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:46.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:46.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:46.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:46.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:46.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:46.362+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:47.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:47.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:47.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:47.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:47.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:47.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:47.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:47.203+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:47.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:47.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:47.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:47.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:47.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:47.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:47.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:47.366+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:47.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:47.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:47.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:47.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:47.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:48.405 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:48.062+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8109 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:48.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:48.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:48.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:48.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:48.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:48.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:48.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:48.189+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:48.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:48.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:48.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:48.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:48.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:48.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:48.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:48.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:48.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:48.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:48.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:48.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:48.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:48.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:48.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:48.383+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:49.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:49.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:49.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:49.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:49.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:49.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:49.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:49.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:49.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:49.348+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:49.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:49.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:49.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:49.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:49.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:49.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:49.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:49.185+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:50.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:50.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:50.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:50.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:50.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:50.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:50.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:50.419+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:50.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:50.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:50.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:50.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:50.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:50.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:50.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:50.371+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:50.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:50.226+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:50.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:50.226+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:50.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:50.226+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:50.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:50.227+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:51.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:51.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:51.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:51.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:51.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:51.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:51.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:51.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:51.559 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:51.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:51.559 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:51.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:51.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:51.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:51.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:51.381+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:51.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:51.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:51.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:51.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:51.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:51.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:51.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:51.251+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:52.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:52.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:52.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:52.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:52.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:52.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:52.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:52.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:52.565 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:52.409+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:52.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:52.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:52.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:52.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:52.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:52.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:52.566 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:52.232+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:53.428 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:53.062+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8205 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:53.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:53.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:53.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:53.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:53.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:53.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:53.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:53.254+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:53.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:53.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:53.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:53.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:53.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:53.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:53.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:53.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:53.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:53.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:53.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:53.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:53.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:53.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:53.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:53.415+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:54.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:54.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:54.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:54.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:54.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:54.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:54.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:54.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:54.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:54.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:54.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:54.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:54.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:54.390+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:54.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:54.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:54.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:54.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:54.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:54.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:54.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:54.274+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:55.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:55.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:55.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:55.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:55.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:55.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:55.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:55.455+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:55.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:55.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:55.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:55.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:55.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:55.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:55.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:55.369+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:55.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:55.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:55.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:55.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:55.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:56.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:56.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:56.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:56.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:56.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:56.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:56.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:56.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:56.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:56.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:56.405+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:56.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:56.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:56.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:56.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:56.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:56.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:56.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:56.287+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:57.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:57.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:57.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:57.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:57.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:57.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:57.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:57.495+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:57.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:57.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:57.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:57.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:57.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:57.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:57.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:57.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:57.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:57.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:57.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:57.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:57.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:57.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:57.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:57.316+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:58.339 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:14:58.064+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8297 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:14:58.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:58.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:58.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:58.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:58.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:58.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:58.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:58.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:58.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:58.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:58.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:58.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:58.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:58.404+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:58.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:58.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:58.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:58.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:58.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:58.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:58.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:58.448+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:59.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:59.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:59.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:59.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:59.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:59.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:14:59.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:14:59.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:14:59.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:59.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:59.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:59.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:59.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:59.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:14:59.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:14:59.376+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:14:59.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:59.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:59.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:59.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:59.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:59.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:14:59.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:14:59.322+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:00.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:00.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:00.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:00.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:00.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:00.510+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:00.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:00.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:00.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:00.301+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:00.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:00.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:00.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:00.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:00.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:00.420+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:01.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:01.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:01.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:01.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:01.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:01.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:01.531+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:01.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:01.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:01.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:01.445+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:01.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:01.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:01.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:01.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:01.297+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:02.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:02.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:02.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:02.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:02.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:02.572+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:02.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:02.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:02.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:02.336+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:02.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:02.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:02.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:02.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:02.490+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:03.367 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:03.066+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8392 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:03.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:03.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:03.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:03.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:03.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:03.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:03.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:03.531+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:03.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:03.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:03.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:03.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:03.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:03.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:03.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:03.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:03.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:03.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:03.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:03.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:03.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:03.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:03.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:03.535+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:04.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:04.557+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:04.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:04.557+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:04.557+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:04.557+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:04.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:04.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:04.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:04.330+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:04.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:04.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:04.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:04.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:04.528+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:05.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:05.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:05.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:05.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:05.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:05.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:05.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:05.508+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:05.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:05.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:05.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:05.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:05.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:05.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:05.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:05.526+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:05.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:05.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:05.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:05.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:05.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:05.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:05.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:05.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:06.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:06.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:06.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:06.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:06.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:06.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:06.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:06.294+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:06.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:06.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:06.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:06.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:06.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:06.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:06.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:06.531+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:06.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:06.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:06.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:06.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:06.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:06.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:06.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:06.477+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:07.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:07.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:07.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:07.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:07.489+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:07.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:07.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:07.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:07.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:07.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:07.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:07.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:07.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:08.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:07.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:08.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:07.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:08.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:07.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:08.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:07.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:08.336 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:08.068+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8485 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:08.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:08.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:08.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:08.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:08.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:08.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:08.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:08.480+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:08.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:08.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:08.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:08.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:08.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:08.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:08.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:08.315+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:09.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:08.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:09.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:08.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:09.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:08.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:09.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:08.588+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:09.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:09.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:09.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:09.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:09.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:09.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:09.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:09.450+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:09.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:09.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:09.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:09.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:09.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:09.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:09.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:09.366+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:09.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:09.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:09.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:09.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:09.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:09.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:09.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:09.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:10.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:10.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:10.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:10.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:10.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:10.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:10.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:10.399+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:10.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:10.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:10.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:10.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:10.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:10.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:10.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:10.558+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:10.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:10.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:10.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:10.439+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:11.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:11.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:11.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:11.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:11.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:11.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:11.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:11.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:11.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:11.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:11.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:11.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:11.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:11.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:11.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:11.530+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:11.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:11.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:11.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:11.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:11.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:11.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:11.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:11.462+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:12.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:12.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:12.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:12.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:12.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:12.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:12.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:12.424+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:12.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:12.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:12.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:12.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:12.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:12.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:12.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:12.567+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:12.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:12.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:12.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:12.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:12.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:12.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:12.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:12.458+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:13.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:13.069+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8580 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:13.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:13.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:13.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:13.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:13.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:13.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:13.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:13.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:13.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:13.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:13.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:13.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:13.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:13.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:13.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:13.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:13.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:13.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:13.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:13.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:13.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:13.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:13.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:13.476+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:14.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:14.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:14.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:14.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:14.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:14.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:14.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:14.470+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:14.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:14.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:14.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:14.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:14.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:14.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:14.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:14.634+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:14.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:14.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:14.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:14.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:14.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:14.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:14.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:14.494+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:15.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:15.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:15.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:15.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:15.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:15.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:15.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:15.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:15.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:15.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:15.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:15.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:15.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:15.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:15.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:15.684+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:15.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:15.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:15.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:15.454+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:16.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:16.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:16.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:16.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:16.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:16.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:16.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:16.467+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:16.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:16.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:16.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:16.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:16.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:16.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:16.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:16.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:16.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:16.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:16.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:16.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:16.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:16.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:16.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:16.635+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:17.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:17.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:17.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:17.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:17.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:17.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:17.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:17.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:17.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:17.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:17.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:17.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:17.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:17.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:17.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:17.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:17.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:17.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:17.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:17.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:17.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:17.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:17.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:17.441+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:18.449 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:18.071+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8672 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:18.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:18.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:18.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:18.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:18.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:18.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:18.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:18.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:18.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:18.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:18.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:18.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:18.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:18.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:18.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:18.636+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:18.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:18.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:18.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:18.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:18.437+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:19.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:19.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:19.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:19.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:19.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:19.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:19.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:19.400+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:19.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:19.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:19.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:19.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:19.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:19.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:19.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:19.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:19.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:19.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:19.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:19.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:19.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:19.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:19.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:19.457+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:20.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:20.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:20.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:20.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:20.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:20.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:20.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:20.421+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:20.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:20.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:20.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:20.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:20.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:20.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:20.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:20.597+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:20.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:20.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:20.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:20.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:20.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:20.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:20.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:20.468+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:21.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:21.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:21.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:21.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:21.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:21.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:21.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:21.423+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:21.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:21.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:21.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:21.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:21.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:21.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:21.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:21.572+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:21.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:21.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:21.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:21.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:21.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:21.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:21.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:21.468+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:22.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:22.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:22.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:22.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:22.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:22.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:22.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:22.469+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:22.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:22.532+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:22.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:22.532+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:22.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:22.532+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:22.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:22.532+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:22.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:22.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:22.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:22.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:22.509+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:23.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:23.072+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8767 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:23.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:23.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:23.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:23.551+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:23.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:23.551+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:23.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:23.551+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:23.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:23.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:23.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:23.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:23.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:23.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:23.547+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:23.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:23.547+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:23.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:23.547+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:23.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:23.547+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:24.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:24.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:24.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:24.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:24.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:24.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:24.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:24.526+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:24.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:24.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:24.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:24.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:24.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:24.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:24.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:24.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:24.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:24.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:24.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:24.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:24.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:24.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:24.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:24.504+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:25.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:25.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:25.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:25.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:25.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:25.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:25.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:25.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:25.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:25.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:25.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:25.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:25.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:25.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:25.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:25.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:25.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:25.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:25.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:25.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:25.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:25.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:25.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:25.573+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:26.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:26.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:26.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:26.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:26.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:26.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:26.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:26.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:26.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:26.531+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:26.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:26.531+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:26.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:26.531+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:26.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:26.532+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:26.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:26.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:26.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:26.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:26.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:26.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:26.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:26.458+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:27.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:27.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:27.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:27.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:27.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:27.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:27.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:27.499+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:27.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:27.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:27.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:27.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:27.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:27.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:27.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:27.485+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:27.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:27.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:27.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:27.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:27.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:27.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:27.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:27.467+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:28.459 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:28.074+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8860 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:28.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:28.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:28.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:28.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:28.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:28.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:28.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:28.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:28.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:28.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:28.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:28.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:28.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:28.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:28.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:28.535+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:28.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:28.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:28.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:28.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:28.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:28.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:28.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:28.440+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:29.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:29.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:29.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:29.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:29.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:29.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:29.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:29.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:29.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:29.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:29.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:29.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:29.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:29.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:29.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:29.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:29.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:29.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:29.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:29.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:29.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:29.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:30.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:30.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:30.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:30.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:30.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:30.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:30.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:30.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:30.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:30.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:30.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:30.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:30.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:30.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:30.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:30.563+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:30.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:30.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:30.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:30.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:30.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:30.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:30.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:30.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:31.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:31.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:31.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:31.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:31.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:31.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:31.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:31.586+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:31.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:31.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:31.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:31.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:31.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:31.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:31.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:31.584+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:31.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:31.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:31.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:31.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:31.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:31.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:31.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:31.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:33.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:32.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:33.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:32.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:33.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:32.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:33.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:32.557+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:33.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:32.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:33.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:32.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:33.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:32.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:33.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:32.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:33.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:32.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:33.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:32.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:33.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:32.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:33.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:32.548+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:33.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:33.076+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 8955 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:33.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:33.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:33.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:33.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:33.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:33.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:33.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:33.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:33.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:33.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:33.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:33.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:33.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:33.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:33.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:33.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:33.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:33.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:33.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:33.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:33.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:33.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:33.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:33.565+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:34.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:34.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:34.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:34.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:34.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:34.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:34.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:34.528+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:34.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:34.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:34.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:34.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:34.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:34.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:34.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:34.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:34.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:34.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:34.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:34.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:34.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:34.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:34.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:34.518+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:35.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:35.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:35.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:35.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:35.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:35.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:35.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:35.553+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:35.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:35.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:35.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:35.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:35.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:35.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:35.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:35.533+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:35.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:35.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:35.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:35.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:35.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:35.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:35.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:35.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:36.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:36.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:36.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:36.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:36.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:36.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:36.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:36.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:36.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:36.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:36.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:36.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:36.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:36.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:36.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:36.563+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:36.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:36.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:36.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:36.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:36.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:36.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:36.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:36.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:37.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:37.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:37.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:37.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:37.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:37.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:37.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:37.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:37.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:37.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:37.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:37.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:37.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:37.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:37.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:37.608+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:37.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:37.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:37.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:37.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:37.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:37.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:37.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:37.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:38.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:38.078+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9047 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:38.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:38.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:38.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:38.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:38.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:38.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:38.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:38.546+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:38.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:38.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:38.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:38.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:38.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:38.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:38.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:38.628+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:38.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:38.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:38.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:38.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:38.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:38.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:38.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:38.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:39.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:39.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:39.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:39.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:39.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:39.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:39.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:39.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:39.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:39.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:39.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:39.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:39.615+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:39.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:39.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:39.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:39.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:39.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:40.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:40.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:40.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:40.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:40.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:40.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:40.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:40.536+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:40.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:40.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:40.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:40.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:40.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:40.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:40.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:40.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:40.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:40.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:40.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:40.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:40.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:40.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:40.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:40.535+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:41.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:41.532+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:41.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:41.532+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:41.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:41.532+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:41.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:41.533+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:41.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:41.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:41.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:41.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:41.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:41.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:41.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:41.642+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:41.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:41.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:41.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:41.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:41.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:41.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:41.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:41.513+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:42.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:42.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:42.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:42.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:42.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:42.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:42.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:42.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:42.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:42.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:42.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:42.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:42.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:42.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:42.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:42.676+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:42.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:42.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:42.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:42.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:42.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:42.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:42.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:42.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:43.455 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:43.080+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9141 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:43.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:43.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:43.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:43.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:43.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:43.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:43.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:43.651+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:43.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:43.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:43.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:43.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:43.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:43.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:43.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:43.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:43.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:43.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:43.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:43.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:43.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:43.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:43.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:43.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:44.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:44.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:44.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:44.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:44.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:44.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:44.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:44.603+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:44.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:44.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:44.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:44.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:44.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:44.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:44.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:44.400+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:44.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:44.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:44.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:44.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:44.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:44.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:44.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:44.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:45.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:45.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:45.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:45.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:45.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:45.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:45.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:45.588+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:45.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:45.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:45.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:45.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:45.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:45.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:45.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:45.583+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:45.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:45.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:45.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:45.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:45.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:45.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:45.812 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:45.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:46.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:46.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:46.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:46.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:46.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:46.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:46.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:46.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:46.600+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:46.600+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:46.600+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:46.601+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:47.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:46.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:47.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:46.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:47.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:46.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:47.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:46.602+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:47.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:47.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:47.608 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:47.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:47.608 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:47.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:47.608 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:47.329+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:48.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:47.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:48.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:47.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:48.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:47.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:48.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:47.588+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:48.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:47.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:48.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:47.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:48.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:47.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:48.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:47.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:48.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:48.082+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9234 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:48.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:48.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:48.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:48.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:48.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:48.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:48.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:48.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:49.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:48.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:49.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:48.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:49.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:48.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:49.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:48.627+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:49.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:48.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:49.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:48.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:49.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:48.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:49.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:48.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:49.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:49.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:49.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:49.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:49.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:49.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:49.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:49.305+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:50.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:49.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:50.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:49.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:50.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:49.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:50.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:49.614+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:50.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:49.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:50.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:49.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:50.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:49.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:50.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:49.616+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:50.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:50.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:50.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:50.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:50.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:50.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:50.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:50.308+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:51.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:50.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:51.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:50.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:51.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:50.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:51.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:50.576+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:51.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:50.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:51.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:50.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:51.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:50.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:51.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:50.630+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:51.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:51.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:51.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:51.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:51.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:51.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:51.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:51.278+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:52.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:51.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:52.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:51.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:52.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:51.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:52.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:51.588+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:52.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:51.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:52.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:51.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:52.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:51.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:52.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:51.646+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:52.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:52.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:52.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:52.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:52.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:52.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:52.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:52.268+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:53.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:52.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:53.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:52.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:53.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:52.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:53.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:52.577+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:53.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:52.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:53.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:52.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:53.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:52.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:53.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:52.600+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:53.560 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:53.083+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9329 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:53.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:53.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:53.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:53.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:53.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:53.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:53.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:53.295+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:54.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:53.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:54.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:53.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:54.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:53.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:54.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:53.619+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:54.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:53.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:54.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:53.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:54.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:53.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:54.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:53.584+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:54.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:54.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:54.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:54.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:54.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:54.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:54.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:54.255+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:55.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:54.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:55.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:54.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:55.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:54.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:55.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:54.580+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:55.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:54.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:55.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:54.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:55.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:54.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:55.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:54.632+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:55.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:55.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:55.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:55.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:55.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:55.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:55.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:55.236+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:56.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:55.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:56.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:55.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:56.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:55.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:56.061 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:55.598+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:56.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:55.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:56.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:55.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:56.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:55.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:56.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:55.659+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:56.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:56.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:56.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:56.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:56.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:56.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:56.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:56.266+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:57.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:56.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:57.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:56.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:57.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:56.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:57.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:56.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:57.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:56.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:57.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:56.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:57.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:56.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:57.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:56.647+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:57.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:57.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:57.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:57.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:57.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:57.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:57.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:57.233+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:58.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:57.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:58.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:57.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:58.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:57.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:58.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:57.563+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:58.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:57.612+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:58.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:57.612+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:58.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:57.612+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:58.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:57.612+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:58.553 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:15:58.085+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9421 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:15:58.553 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:58.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:58.553 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:58.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:58.553 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:58.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:58.553 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:58.219+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:58.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:58.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:58.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:58.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:58.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:58.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:58.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:58.534+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:58.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:58.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:58.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:58.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:58.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:58.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:58.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:58.642+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:15:59.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:59.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:59.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:59.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:59.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:59.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:15:59.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:15:59.204+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:59.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:59.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:59.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:59.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:59.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:59.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:15:59.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:15:59.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:15:59.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:59.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:59.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:59.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:59.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:59.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:15:59.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:15:59.615+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:00.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:00.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:00.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:00.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:00.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:00.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:00.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:00.178+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:00.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:00.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:00.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:00.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:00.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:00.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:00.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:00.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:00.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:00.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:00.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:00.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:00.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:00.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:00.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:00.579+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:01.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:01.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:01.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:01.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:01.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:01.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:01.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:01.227+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:02.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:01.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:02.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:01.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:02.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:01.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:02.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:01.552+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:02.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:01.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:02.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:01.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:02.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:01.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:02.061 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:01.596+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:02.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:02.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:02.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:02.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:02.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:02.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:02.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:02.238+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:02.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:02.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:02.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:02.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:02.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:02.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:02.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:02.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:02.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:02.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:02.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:02.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:02.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:02.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:02.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:02.639+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:03.505 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:03.087+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9515 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:03.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:03.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:03.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:03.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:03.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:03.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:03.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:03.201+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:03.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:03.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:03.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:03.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:03.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:03.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:03.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:03.640+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:03.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:03.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:03.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:03.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:03.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:03.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:03.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:03.487+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:04.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:04.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:04.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:04.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:04.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:04.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:04.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:04.211+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:04.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:04.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:04.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:04.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:04.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:04.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:04.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:04.688+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:04.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:04.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:04.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:04.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:04.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:04.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:04.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:04.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:05.471 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:05.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:05.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:05.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:05.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:05.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:05.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:05.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:05.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:05.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:05.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:05.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:05.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:05.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:05.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:05.660+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:05.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:05.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:05.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:05.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:05.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:05.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:05.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:05.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:06.451 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:06.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:06.451 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:06.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:06.451 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:06.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:06.452 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:06.180+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:06.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:06.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:06.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:06.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:06.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:06.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:06.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:06.433+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:06.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:06.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:06.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:06.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:06.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:06.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:06.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:06.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:07.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:07.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:07.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:07.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:07.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:07.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:07.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:07.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:07.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:07.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:07.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:07.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:07.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:07.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:07.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:07.167+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:08.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:07.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:08.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:07.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:08.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:07.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:08.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:07.648+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:08.363 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:08.088+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9608 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:08.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:08.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:08.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:08.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:08.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:08.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:08.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:08.200+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:08.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:08.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:08.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:08.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:08.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:08.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:08.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:08.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:09.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:08.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:09.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:08.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:09.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:08.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:09.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:08.642+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:09.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:09.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:09.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:09.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:09.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:09.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:09.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:09.348+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:09.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:09.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:09.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:09.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:09.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:09.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:09.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:09.213+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:09.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:09.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:09.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:09.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:09.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:09.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:09.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:09.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:10.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:10.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:10.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:10.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:10.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:10.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:10.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:10.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:10.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:10.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:10.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:10.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:10.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:10.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:10.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:10.256+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:11.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:10.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:11.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:10.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:11.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:10.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:11.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:10.641+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:11.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:11.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:11.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:11.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:11.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:11.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:11.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:11.389+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:11.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:11.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:11.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:11.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:11.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:11.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:11.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:11.282+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:12.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:11.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:12.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:11.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:12.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:11.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:12.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:11.599+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:12.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:12.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:12.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:12.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:12.419+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:12.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:12.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:12.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:12.329+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:12.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:12.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:12.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:12.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:12.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:12.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:12.570+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:13.431 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:13.090+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9703 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:13.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:13.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:13.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:13.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:13.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:13.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:13.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:13.305+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:13.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:13.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:13.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:13.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:13.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:13.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:13.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:13.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:13.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:13.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:13.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:13.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:13.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:13.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:13.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:13.577+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:14.582 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:14.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:14.583 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:14.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:14.583 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:14.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:14.583 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:14.437+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:14.583 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:14.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:14.583 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:14.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:14.583 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:14.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:14.583 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:14.312+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:15.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:14.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:15.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:14.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:15.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:14.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:15.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:14.564+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:15.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:15.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:15.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:15.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:15.482+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:15.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:15.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:15.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:15.522+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:15.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:15.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:15.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:15.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:15.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:15.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:15.315+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:16.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:16.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:16.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:16.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:16.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:16.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:16.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:16.522+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:16.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:16.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:16.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:16.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:16.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:16.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:16.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:16.522+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:16.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:16.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:16.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:16.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:16.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:16.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:16.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:16.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:17.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:17.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:17.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:17.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:17.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:17.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:17.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:17.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:17.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:17.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:17.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:17.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:17.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:17.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:17.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:17.499+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:17.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:17.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:17.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:17.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:17.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:17.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:17.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:17.342+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:18.397 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:18.092+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9789 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:18.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:18.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:18.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:18.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:18.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:18.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:18.478+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:18.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:18.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:18.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:18.571+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:18.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:18.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:18.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:18.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:18.379+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:19.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:19.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:19.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:19.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:19.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:19.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:19.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:19.567+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:19.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:19.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:19.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:19.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:19.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:19.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:19.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:19.491+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:19.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:19.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:19.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:19.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:19.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:19.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:19.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:19.407+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:20.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:20.447+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:20.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:20.447+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:20.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:20.447+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:20.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:20.448+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:20.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:20.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:20.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:20.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:20.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:20.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:20.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:20.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:20.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:20.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:20.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:20.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:20.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:20.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:20.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:20.594+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:21.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:21.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:21.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:21.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:21.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:21.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:21.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:21.560+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:21.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:21.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:21.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:21.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:21.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:21.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:21.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:21.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:21.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:21.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:21.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:21.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:21.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:21.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:21.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:21.403+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:22.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:22.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:22.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:22.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:22.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:22.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:22.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:22.443+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:22.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:22.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:22.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:22.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:22.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:22.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:22.540+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:22.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:22.540+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:22.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:22.540+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:22.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:22.541+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:23.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:23.094+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9877 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:23.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:23.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:23.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:23.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:23.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:23.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:23.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:23.590+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:23.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:23.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:23.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:23.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:23.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:23.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:23.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:23.487+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:23.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:23.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:23.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:23.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:23.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:23.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:23.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:23.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:24.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:24.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:24.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:24.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:24.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:24.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:24.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:24.587+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:24.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:24.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:24.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:24.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:24.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:24.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:24.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:24.601+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:24.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:24.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:24.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:24.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:24.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:24.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:24.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:24.525+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:25.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:25.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:25.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:25.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:25.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:25.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:25.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:25.546+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:25.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:25.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:25.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:25.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:25.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:25.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:25.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:25.503+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:25.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:25.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:25.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:25.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:25.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:25.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:25.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:25.638+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:26.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:26.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:26.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:26.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:26.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:26.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:26.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:26.576+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:26.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:26.466+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:26.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:26.466+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:26.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:26.466+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:26.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:26.467+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:26.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:26.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:26.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:26.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:26.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:26.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:26.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:26.682+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:27.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:27.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:27.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:27.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:27.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:27.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:27.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:27.600+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:27.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:27.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:27.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:27.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:27.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:27.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:27.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:27.693+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:27.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:27.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:27.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:27.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:27.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:27.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:27.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:27.508+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:28.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:28.096+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 9960 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:28.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:28.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:28.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:28.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:28.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:28.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:28.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:28.495+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:28.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:28.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:28.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:28.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:28.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:28.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:28.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:28.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:28.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:28.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:28.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:28.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:28.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:28.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:28.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:28.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:29.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:29.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:29.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:29.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:29.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:29.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:29.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:29.638+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:29.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:29.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:29.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:29.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:29.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:29.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:29.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:29.510+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:29.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:29.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:29.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:29.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:29.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:29.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:29.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:29.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:30.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:30.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:30.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:30.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:30.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:30.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:30.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:30.626+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:30.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:30.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:30.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:30.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:30.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:30.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:30.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:30.537+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:30.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:30.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:30.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:30.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:30.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:30.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:30.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:30.660+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:31.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:31.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:31.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:31.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:31.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:31.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:31.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:31.652+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:31.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:31.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:31.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:31.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:31.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:31.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:31.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:31.536+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:31.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:31.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:31.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:31.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:31.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:31.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:31.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:31.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:33.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:33.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:33.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:33.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:32.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:33.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:32.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:33.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:32.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:33.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:32.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:33.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:32.573+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:33.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:32.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:33.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:32.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:33.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:32.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:33.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:32.667+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:33.559 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:33.097+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10047 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:34.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:33.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:34.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:33.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:34.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:33.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:34.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:33.579+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:34.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:33.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:34.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:33.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:34.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:33.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:34.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:33.670+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:34.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:33.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:34.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:33.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:34.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:33.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:34.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:33.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:35.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:35.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:35.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:35.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:34.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:35.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:34.602+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:35.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:34.602+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:35.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:34.602+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:35.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:34.603+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:35.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:34.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:35.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:34.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:35.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:34.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:35.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:34.720+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:36.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:35.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:36.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:35.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:36.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:35.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:36.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:35.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:36.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:35.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:36.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:35.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:36.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:35.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:36.060 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:35.597+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:36.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:35.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:36.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:35.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:36.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:35.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:36.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:35.749+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:37.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:36.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:37.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:36.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:37.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:36.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:37.059 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:36.549+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:37.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:37.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:37.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:37.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:36.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:37.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:36.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:37.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:36.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:37.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:36.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:37.060 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:36.654+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:37.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:37.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:37.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:37.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:37.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:37.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:37.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:37.517+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:37.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:37.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:37.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:37.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:37.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:37.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:37.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:37.714+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:37.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:37.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:37.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:37.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:37.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:37.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:37.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:37.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:38.500 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:38.099+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10137 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:38.808 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:38.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:38.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:38.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:38.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:38.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:38.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:38.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:38.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:38.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:38.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:38.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:38.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:38.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:38.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:38.481+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:38.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:38.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:38.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:38.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:38.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:38.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:38.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:38.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:39.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:39.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:39.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:39.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:39.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:39.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:39.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:39.529+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:39.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:39.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:39.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:39.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:39.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:39.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:39.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:39.732+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:39.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:39.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:39.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:39.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:39.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:39.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:39.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:39.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:40.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:40.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:40.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:40.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:40.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:40.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:40.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:40.514+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:40.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:40.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:40.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:40.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:40.734+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:40.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:40.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:40.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:40.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:40.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:40.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:40.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:40.666+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:41.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:41.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:41.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:41.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:41.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:41.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:41.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:41.478+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:41.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:41.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:41.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:41.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:41.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:41.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:41.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:41.700+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:42.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:41.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:42.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:41.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:42.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:41.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:42.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:41.761+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:42.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:42.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:42.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:42.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:42.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:42.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:42.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:42.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:43.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:42.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:43.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:42.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:43.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:42.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:43.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:42.768+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:43.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:42.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:43.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:42.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:43.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:42.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:43.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:42.713+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:43.508 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:43.101+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10230 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:43.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:43.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:43.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:43.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:43.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:43.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:43.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:43.700+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:43.801 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:43.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:43.801 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:43.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:43.801 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:43.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:43.801 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:43.491+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:44.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:43.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:44.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:43.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:44.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:43.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:44.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:43.783+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:44.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:44.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:44.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:44.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:44.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:44.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:44.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:44.681+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:44.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:44.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:44.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:44.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:44.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:44.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:44.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:44.470+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:45.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:44.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:45.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:44.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:45.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:44.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:45.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:44.757+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:45.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:45.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:45.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:45.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:45.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:45.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:45.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:45.426+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:46.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:45.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:46.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:45.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:46.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:45.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:46.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:45.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:46.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:45.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:46.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:45.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:46.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:45.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:46.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:45.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:46.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:46.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:46.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:46.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:46.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:46.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:46.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:46.419+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:47.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:46.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:47.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:46.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:47.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:46.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:47.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:46.681+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:47.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:46.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:47.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:46.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:47.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:46.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:47.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:46.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:47.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:47.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:47.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:47.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:47.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:47.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:47.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:47.377+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:48.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:47.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:48.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:47.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:48.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:47.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:48.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:47.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:48.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:47.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:48.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:47.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:48.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:47.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:48.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:47.743+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:48.144 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:48.103+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10324 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:48.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:48.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:48.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:48.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:48.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:48.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:48.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:48.426+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:49.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:48.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:49.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:48.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:49.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:48.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:49.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:48.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:49.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:48.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:49.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:48.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:49.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:48.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:49.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:48.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:49.744 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:49.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:49.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:49.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:49.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:49.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:49.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:49.401+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:50.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:49.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:50.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:49.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:50.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:49.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:50.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:49.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:50.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:49.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:50.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:49.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:50.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:49.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:50.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:49.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:50.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:50.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:50.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:50.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:50.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:50.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:50.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:50.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:51.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:50.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:51.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:50.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:51.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:50.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:51.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:50.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:51.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:50.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:51.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:50.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:51.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:50.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:51.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:50.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:51.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:51.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:51.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:51.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:51.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:51.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:51.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:51.319+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:52.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:51.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:52.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:51.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:52.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:51.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:52.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:51.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:52.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:51.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:52.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:51.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:52.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:51.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:52.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:51.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:52.807 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:52.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:52.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:52.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:52.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:52.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:52.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:52.312+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:53.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:52.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:53.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:52.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:53.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:52.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:53.059 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:52.798+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:53.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:52.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:53.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:52.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:53.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:52.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:53.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:52.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:53.558 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:53.105+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10420 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:53.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:53.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:53.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:53.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:53.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:53.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:53.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:53.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:53.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:53.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:53.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:53.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:53.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:53.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:53.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:53.772+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:54.274 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:53.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:54.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:53.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:54.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:53.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:54.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:53.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:54.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:54.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:54.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:54.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:54.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:54.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:54.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:54.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:54.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:54.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:54.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:54.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:54.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:54.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:54.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:54.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:55.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:54.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:55.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:54.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:55.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:54.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:55.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:54.880+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:55.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:55.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:55.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:55.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:55.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:55.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:55.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:55.266+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:56.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:55.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:56.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:55.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:56.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:55.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:56.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:55.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:56.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:55.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:56.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:55.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:56.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:55.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:56.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:55.927+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:56.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:56.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:56.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:56.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:56.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:56.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:56.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:56.281+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:56.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:56.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:56.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:56.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:56.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:56.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:56.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:56.722+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:57.298 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:56.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:57.298 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:56.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:57.299 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:56.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:57.299 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:56.975+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:57.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:57.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:57.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:57.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:57.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:57.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:57.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:57.282+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:58.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:57.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:58.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:57.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:58.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:57.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:58.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:57.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:58.308 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:16:58.107+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10513 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:16:58.308 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:58.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:58.308 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:58.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:58.308 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:58.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:58.308 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:58.262+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:58.309 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:58.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:58.309 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:58.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:58.309 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:58.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:58.309 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:58.024+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:59.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:58.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:59.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:58.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:59.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:58.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:59.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:58.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:59.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:59.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:59.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:59.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:59.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:59.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:16:59.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:16:59.245+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:16:59.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:59.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:59.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:59.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:59.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:59.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:16:59.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:16:59.068+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:16:59.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:59.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:59.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:59.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:59.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:59.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:16:59.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:16:59.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:00.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:00.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:00.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:00.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:00.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:00.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:00.559 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:00.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:00.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:00.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:00.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:00.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:00.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:00.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:00.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:00.049+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:01.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:00.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:01.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:00.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:01.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:00.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:01.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:00.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:01.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:01.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:01.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:01.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:01.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:01.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:01.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:01.029+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:01.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:01.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:01.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:01.317+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:02.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:01.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:02.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:01.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:02.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:01.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:02.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:01.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:02.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:02.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:02.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:02.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:02.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:02.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:02.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:02.050+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:02.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:02.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:02.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:02.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:02.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:02.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:02.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:02.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:03.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:02.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:03.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:02.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:03.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:02.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:03.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:02.756+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:03.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:03.110+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10608 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:03.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:03.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:03.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:03.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:03.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:03.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:03.402 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:03.093+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:03.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:03.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:03.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:03.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:03.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:03.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:03.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:03.385+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:04.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:03.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:04.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:03.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:04.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:03.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:04.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:03.789+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:04.379 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:04.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:04.379 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:04.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:04.379 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:04.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:04.380 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:04.059+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:04.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:04.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:04.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:04.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:04.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:04.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:04.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:04.363+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:05.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:04.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:05.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:04.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:05.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:04.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:05.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:04.750+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:05.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:05.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:05.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:05.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:05.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:05.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:05.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:05.014+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:05.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:05.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:05.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:05.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:05.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:05.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:05.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:05.363+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:06.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:05.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:06.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:05.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:06.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:05.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:06.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:05.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:06.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:05.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:06.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:05.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:06.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:05.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:06.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:05.973+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:06.714 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:06.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:06.714 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:06.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:06.714 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:06.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:06.714 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:06.356+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:07.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:06.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:07.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:06.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:07.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:06.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:07.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:06.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:07.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:07.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:07.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:07.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:07.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:07.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:07.308 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:07.020+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:07.725 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:07.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:07.725 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:07.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:07.725 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:07.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:07.725 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:07.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:08.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:07.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:08.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:07.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:08.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:07.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:08.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:07.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:08.365 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:08.112+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10702 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:08.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:08.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:08.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:08.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:08.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:08.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:08.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:08.065+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:08.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:08.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:08.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:08.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:08.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:08.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:08.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:08.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:09.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:08.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:09.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:08.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:09.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:08.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:09.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:08.735+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:09.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:09.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:09.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:09.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:09.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:09.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:09.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:09.112+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:09.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:09.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:09.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:09.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:09.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:09.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:09.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:09.395+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:10.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:09.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:10.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:09.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:10.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:09.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:10.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:09.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:10.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:10.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:10.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:10.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:10.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:10.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:10.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:10.126+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:10.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:10.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:10.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:10.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:10.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:10.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:10.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:10.678+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:10.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:10.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:10.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:10.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:10.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:10.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:10.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:10.439+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:11.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:11.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:11.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:11.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:11.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:11.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:11.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:11.110+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:11.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:11.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:11.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:11.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:11.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:11.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:11.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:11.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:12.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:11.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:12.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:11.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:12.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:11.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:12.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:11.664+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:12.395 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:12.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:12.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:12.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:12.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:12.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:12.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:12.063+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:12.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:12.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:12.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:12.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:12.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:12.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:12.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:12.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:13.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:12.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:13.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:12.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:13.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:12.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:13.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:12.641+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:13.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:13.114+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10797 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:13.368 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:13.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:13.368 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:13.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:13.368 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:13.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:13.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:13.083+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:13.807 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:13.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:13.807 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:13.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:13.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:13.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:13.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:13.352+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:13.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:13.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:13.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:13.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:13.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:13.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:13.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:13.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:14.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:14.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:14.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:14.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:14.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:14.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:14.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:14.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:14.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:14.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:14.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:14.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:14.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:14.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:14.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:14.070+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:15.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:14.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:15.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:14.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:15.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:14.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:15.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:14.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:15.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:15.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:15.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:15.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:15.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:15.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:15.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:15.322+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:15.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:15.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:15.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:15.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:15.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:15.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:15.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:15.116+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:16.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:15.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:16.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:15.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:16.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:15.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:16.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:15.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:16.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:16.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:16.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:16.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:16.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:16.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:16.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:16.344+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:16.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:16.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:16.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:16.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:16.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:16.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:16.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:16.116+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:17.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:16.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:17.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:16.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:17.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:16.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:17.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:16.575+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:17.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:17.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:17.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:17.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:17.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:17.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:17.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:17.318+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:17.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:17.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:17.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:17.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:17.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:17.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:17.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:17.091+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:18.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:17.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:18.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:17.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:18.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:17.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:18.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:17.606+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:18.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:18.116+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10889 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:18.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:18.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:18.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:18.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:18.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:18.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:18.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:18.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:18.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:18.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:18.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:18.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:18.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:18.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:18.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:18.116+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:19.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:18.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:19.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:18.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:19.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:18.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:19.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:18.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:19.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:19.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:19.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:19.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:19.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:19.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:19.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:19.118+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:19.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:19.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:19.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:19.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:19.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:19.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:19.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:19.323+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:19.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:19.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:19.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:19.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:19.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:19.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:19.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:19.568+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:20.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:20.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:20.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:20.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:20.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:20.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:20.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:20.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:20.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:20.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:20.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:20.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:20.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:20.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:20.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:20.108+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:21.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:20.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:21.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:20.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:21.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:20.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:21.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:20.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:21.554 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:21.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:21.554 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:21.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:21.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:21.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:21.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:21.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:21.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:21.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:21.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:21.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:21.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:21.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:21.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:21.139+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:21.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:21.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:21.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:21.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:21.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:21.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:21.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:21.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:22.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:22.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:22.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:22.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:22.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:22.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:22.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:22.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:22.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:22.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:22.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:22.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:22.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:22.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:22.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:22.121+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:23.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:22.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:23.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:22.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:23.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:22.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:23.058 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:22.578+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:23.410 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:23.118+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 10982 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:23.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:23.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:23.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:23.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:23.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:23.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:23.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:23.098+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:23.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:23.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:23.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:23.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:23.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:23.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:23.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:23.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:23.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:23.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:23.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:23.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:23.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:23.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:23.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:23.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:24.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:24.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:24.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:24.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:24.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:24.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:24.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:24.076+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:24.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:24.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:24.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:24.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:24.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:24.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:24.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:24.599+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:24.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:24.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:24.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:24.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:24.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:24.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:24.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:24.375+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:25.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:25.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:25.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:25.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:25.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:25.373+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:25.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:25.374+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:25.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:25.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:25.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:25.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:25.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:25.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:25.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:25.126+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:26.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:25.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:26.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:25.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:26.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:25.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:26.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:25.602+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:26.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:26.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:26.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:26.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:26.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:26.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:26.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:26.124+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:26.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:26.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:26.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:26.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:26.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:26.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:26.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:26.620+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:26.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:26.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:26.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:26.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:26.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:26.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:26.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:26.410+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:27.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:27.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:27.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:27.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:27.560 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:27.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:27.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:27.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:27.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:27.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:27.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:27.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:27.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:27.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:27.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:27.154+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:28.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:27.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:28.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:27.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:28.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:27.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:28.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:27.602+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:28.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:28.120+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11076 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:28.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:28.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:28.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:28.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:28.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:28.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:28.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:28.171+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:28.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:28.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:28.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:28.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:28.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:28.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:28.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:28.364+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:29.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:28.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:29.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:28.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:29.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:28.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:29.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:28.615+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:29.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:29.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:29.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:29.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:29.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:29.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:29.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:29.345+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:29.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:29.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:29.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:29.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:29.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:29.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:29.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:29.164+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:29.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:29.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:29.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:29.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:29.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:29.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:29.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:29.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:30.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:30.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:30.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:30.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:30.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:30.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:30.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:30.306+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:30.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:30.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:30.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:30.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:30.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:30.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:30.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:30.195+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:31.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:30.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:31.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:30.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:31.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:30.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:31.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:30.598+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:31.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:31.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:31.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:31.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:31.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:31.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:31.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:31.281+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:31.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:31.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:31.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:31.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:31.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:31.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:31.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:31.187+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:32.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:31.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:32.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:31.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:32.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:31.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:32.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:31.610+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:32.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:32.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:32.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:32.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:32.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:32.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:32.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:32.148+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:32.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:32.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:32.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:32.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:32.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:32.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:32.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:32.273+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:33.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:32.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:33.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:32.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:33.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:32.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:33.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:32.660+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:33.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:33.122+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11169 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:33.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:33.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:33.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:33.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:33.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:33.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:33.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:33.288+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:33.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:33.107+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:33.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:33.107+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:33.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:33.107+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:33.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:33.107+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:34.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:33.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:34.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:33.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:34.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:33.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:34.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:33.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:34.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:34.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:34.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:34.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:34.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:34.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:34.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:34.118+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:34.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:34.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:34.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:34.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:34.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:34.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:34.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:34.309+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:35.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:34.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:35.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:34.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:35.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:34.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:35.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:34.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:35.311 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:35.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:35.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:35.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:35.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:35.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:35.312 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:35.162+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:35.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:35.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:35.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:35.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:35.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:35.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:35.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:35.296+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:36.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:35.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:36.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:35.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:36.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:35.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:36.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:35.676+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:36.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:36.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:36.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:36.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:36.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:36.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:36.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:36.336+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:36.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:36.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:36.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:36.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:36.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:36.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:36.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:36.191+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:37.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:36.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:37.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:36.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:37.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:36.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:37.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:36.699+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:37.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:37.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:37.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:37.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:37.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:37.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:37.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:37.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:37.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:37.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:37.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:37.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:37.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:37.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:37.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:37.203+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:38.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:37.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:38.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:37.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:38.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:37.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:38.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:37.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:38.384 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:38.124+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11261 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:38.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:38.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:38.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:38.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:38.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:38.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:38.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:38.175+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:38.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:38.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:38.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:38.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:38.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:38.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:38.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:38.369+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:39.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:38.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:39.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:38.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:39.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:38.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:39.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:38.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:39.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:39.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:39.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:39.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:39.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:39.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:39.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:39.330+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:39.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:39.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:39.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:39.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:39.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:39.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:39.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:39.166+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:40.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:39.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:40.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:39.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:40.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:39.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:40.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:39.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:40.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:40.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:40.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:40.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:40.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:40.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:40.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:40.325+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:40.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:40.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:40.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:40.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:40.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:40.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:40.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:40.132+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:41.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:40.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:41.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:40.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:41.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:40.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:41.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:40.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:41.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:41.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:41.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:41.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:41.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:41.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:41.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:41.285+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:41.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:41.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:41.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:41.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:41.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:41.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:41.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:41.108+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:42.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:41.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:42.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:41.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:42.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:41.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:42.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:41.748+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:42.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:42.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:42.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:42.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:42.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:42.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:42.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:42.291+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:42.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:42.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:42.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:42.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:42.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:42.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:42.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:42.097+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:43.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:42.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:43.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:42.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:43.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:42.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:43.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:42.700+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:43.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:43.126+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11353 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:43.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:43.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:43.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:43.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:43.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:43.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:43.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:43.064+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:43.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:43.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:43.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:43.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:43.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:43.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:43.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:43.259+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:44.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:43.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:44.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:43.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:44.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:43.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:44.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:43.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:44.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:44.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:44.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:44.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:44.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:44.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:44.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:44.098+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:44.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:44.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:44.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:44.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:44.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:44.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:44.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:44.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:45.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:44.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:45.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:44.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:45.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:44.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:45.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:44.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:45.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:45.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:45.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:45.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:45.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:45.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:45.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:45.135+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:45.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:45.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:45.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:45.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:45.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:45.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:45.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:45.233+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:46.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:45.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:46.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:45.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:46.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:45.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:46.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:45.758+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:46.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:46.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:46.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:46.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:46.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:46.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:46.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:46.245+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:46.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:46.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:46.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:46.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:46.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:46.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:46.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:46.107+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:47.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:46.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:47.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:46.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:47.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:46.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:47.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:46.765+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:47.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:47.144+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:47.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:47.144+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:47.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:47.144+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:47.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:47.145+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:47.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:47.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:47.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:47.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:47.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:47.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:47.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:47.219+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:48.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:47.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:48.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:47.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:48.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:47.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:48.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:47.743+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:48.557 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:48.128+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11445 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:48.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:48.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:48.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:48.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:48.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:48.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:48.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:48.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:48.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:48.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:48.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:48.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:48.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:48.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:48.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:48.171+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:49.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:48.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:49.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:48.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:49.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:48.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:49.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:48.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:49.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:49.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:49.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:49.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:49.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:49.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:49.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:49.141+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:49.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:49.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:49.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:49.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:49.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:49.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:49.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:49.222+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:49.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:49.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:49.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:49.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:49.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:49.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:49.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:49.715+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:50.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:50.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:50.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:50.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:50.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:50.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:50.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:50.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:50.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:50.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:50.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:50.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:50.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:50.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:50.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:50.177+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:51.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:50.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:51.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:50.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:51.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:50.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:51.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:50.703+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:51.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:51.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:51.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:51.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:51.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:51.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:51.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:51.276+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:51.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:51.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:51.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:51.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:51.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:51.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:51.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:51.219+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:52.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:51.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:52.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:51.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:52.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:51.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:52.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:51.673+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:52.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:52.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:52.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:52.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:52.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:52.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:52.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:52.302+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:52.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:52.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:52.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:52.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:52.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:52.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:52.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:52.171+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:52.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:52.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:52.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:52.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:52.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:52.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:52.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:52.700+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:53.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:53.130+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11539 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:53.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:53.181+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:53.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:53.181+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:53.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:53.181+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:53.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:53.182+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:53.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:53.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:53.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:53.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:53.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:53.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:53.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:53.269+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:54.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:53.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:54.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:53.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:54.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:53.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:54.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:53.690+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:54.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:54.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:54.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:54.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:54.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:54.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:54.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:54.197+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:54.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:54.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:54.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:54.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:54.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:54.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:54.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:54.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:55.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:54.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:55.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:54.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:55.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:54.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:55.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:54.663+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:55.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:55.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:55.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:55.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:55.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:55.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:55.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:55.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:55.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:55.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:55.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:55.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:55.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:55.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:55.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:55.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:56.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:55.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:56.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:55.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:56.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:55.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:56.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:55.623+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:56.348 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:56.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:56.348 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:56.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:56.349 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:56.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:56.349 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:56.157+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:56.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:56.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:56.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:56.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:56.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:56.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:56.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:56.333+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:57.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:56.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:57.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:56.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:57.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:56.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:57.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:56.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:57.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:57.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:57.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:57.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:57.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:57.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:57.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:57.375+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:57.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:57.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:57.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:57.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:57.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:57.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:57.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:57.157+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:58.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:57.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:58.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:57.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:58.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:57.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:58.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:57.581+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:58.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:17:58.132+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11632 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:17:58.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:58.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:58.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:58.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:58.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:58.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:58.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:58.362+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:58.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:58.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:58.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:58.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:58.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:58.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:58.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:58.164+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:59.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:58.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:59.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:58.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:59.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:58.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:59.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:58.602+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:59.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:59.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:59.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:59.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:59.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:59.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:17:59.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:17:59.210+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:17:59.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:59.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:59.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:59.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:59.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:59.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:17:59.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:17:59.344+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:17:59.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:59.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:59.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:59.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:59.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:59.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:17:59.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:17:59.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:00.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:00.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:00.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:00.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:00.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:00.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:00.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:00.300+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:00.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:00.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:00.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:00.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:00.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:00.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:00.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:00.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:01.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:00.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:01.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:00.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:01.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:00.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:01.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:00.553+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:01.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:01.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:01.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:01.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:01.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:01.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:01.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:01.261+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:01.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:01.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:01.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:01.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:01.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:01.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:01.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:01.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:02.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:01.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:02.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:01.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:02.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:01.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:02.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:01.589+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:02.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:02.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:02.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:02.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:02.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:02.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:02.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:02.300+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:02.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:02.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:02.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:02.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:02.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:02.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:02.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:02.279+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:03.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:02.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:03.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:02.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:03.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:02.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:03.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:02.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:03.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:03.134+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11724 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:03.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:03.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:03.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:03.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:03.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:03.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:03.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:03.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:03.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:03.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:03.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:03.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:03.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:03.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:03.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:03.305+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:04.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:03.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:04.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:03.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:04.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:03.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:04.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:03.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:04.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:04.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:04.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:04.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:04.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:04.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:04.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:04.353+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:04.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:04.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:04.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:04.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:04.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:04.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:04.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:04.309+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:05.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:04.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:05.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:04.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:05.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:04.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:05.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:04.648+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:05.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:05.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:05.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:05.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:05.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:05.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:05.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:05.359+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:05.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:05.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:05.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:05.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:05.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:05.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:05.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:05.316+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:06.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:05.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:06.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:05.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:06.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:05.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:06.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:05.693+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:06.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:06.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:06.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:06.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:06.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:06.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:06.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:06.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:06.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:06.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:06.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:06.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:06.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:06.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:06.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:06.306+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:07.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:06.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:07.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:06.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:07.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:06.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:07.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:06.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:07.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:07.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:07.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:07.364+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:07.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:07.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:07.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:07.325+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:08.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:07.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:08.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:07.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:08.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:07.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:08.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:07.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:08.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:08.137+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11818 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:08.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:08.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:08.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:08.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:08.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:08.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:08.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:08.353+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:08.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:08.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:08.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:08.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:08.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:08.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:08.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:08.315+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:09.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:08.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:09.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:08.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:09.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:08.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:09.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:08.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:09.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:09.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:09.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:09.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:09.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:09.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:09.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:09.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:09.719 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:09.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:09.719 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:09.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:09.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:09.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:09.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:09.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:10.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:09.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:10.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:09.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:10.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:09.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:10.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:09.705+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:10.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:10.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:10.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:10.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:10.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:10.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:10.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:10.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:10.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:10.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:10.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:10.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:10.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:10.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:10.731 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:10.392+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:11.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:10.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:11.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:10.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:11.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:10.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:11.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:10.717+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:11.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:11.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:11.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:11.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:11.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:11.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:11.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:11.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:11.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:11.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:11.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:11.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:11.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:11.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:11.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:11.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:12.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:11.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:12.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:11.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:12.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:11.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:12.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:11.740+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:12.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:12.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:12.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:12.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:12.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:12.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:12.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:12.337+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:12.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:12.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:12.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:12.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:12.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:12.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:12.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:12.259+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:13.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:12.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:13.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:12.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:13.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:12.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:13.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:12.715+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:13.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:13.139+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 11912 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:13.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:13.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:13.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:13.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:13.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:13.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:13.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:13.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:13.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:13.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:13.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:13.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:13.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:13.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:13.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:13.301+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:14.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:13.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:14.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:13.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:14.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:13.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:14.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:13.683+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:14.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:14.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:14.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:14.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:14.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:14.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:14.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:14.248+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:14.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:14.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:14.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:14.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:14.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:14.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:14.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:14.289+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:15.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:14.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:15.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:14.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:15.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:14.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:15.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:14.713+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:15.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:15.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:15.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:15.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:15.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:15.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:15.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:15.247+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:15.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:15.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:15.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:15.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:15.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:15.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:15.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:15.262+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:16.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:15.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:16.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:15.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:16.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:15.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:16.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:15.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:16.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:16.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:16.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:16.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:16.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:16.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:16.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:16.288+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:16.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:16.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:16.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:16.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:16.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:16.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:16.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:16.283+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:17.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:16.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:17.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:16.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:17.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:16.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:17.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:16.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:17.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:17.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:17.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:17.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:17.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:17.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:17.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:17.316+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:17.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:17.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:17.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:17.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:17.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:17.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:17.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:17.240+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:18.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:17.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:18.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:17.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:18.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:17.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:18.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:17.764+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:18.555 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:18.141+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12006 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:18.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:18.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:18.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:18.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:18.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:18.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:18.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:18.270+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:18.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:18.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:18.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:18.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:18.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:18.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:18.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:18.293+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:19.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:18.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:19.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:18.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:19.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:18.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:19.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:18.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:19.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:19.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:19.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:19.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:19.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:19.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:19.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:19.312+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:19.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:19.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:19.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:19.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:19.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:19.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:19.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:19.233+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:19.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:19.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:19.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:19.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:19.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:19.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:19.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:19.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:20.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:20.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:20.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:20.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:20.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:20.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:20.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:20.262+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:20.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:20.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:20.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:20.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:20.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:20.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:20.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:20.277+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:21.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:20.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:21.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:20.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:21.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:20.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:21.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:20.719+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:21.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:21.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:21.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:21.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:21.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:21.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:21.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:21.263+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:21.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:21.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:21.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:21.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:21.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:21.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:21.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:21.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:22.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:21.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:22.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:21.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:22.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:21.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:22.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:21.759+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:22.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:22.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:22.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:22.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:22.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:22.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:22.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:22.230+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:22.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:22.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:22.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:22.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:22.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:22.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:22.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:22.322+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:22.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:22.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:22.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:22.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:23.555 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:23.143+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12099 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:23.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:23.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:23.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:23.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:23.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:23.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:23.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:23.264+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:23.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:23.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:23.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:23.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:23.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:23.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:23.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:23.354+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:24.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:23.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:24.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:23.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:24.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:23.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:24.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:23.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:24.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:24.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:24.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:24.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:24.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:24.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:24.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:24.371+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:24.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:24.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:24.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:24.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:24.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:24.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:24.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:24.299+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:25.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:24.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:25.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:24.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:25.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:24.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:25.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:24.765+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:25.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:25.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:25.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:25.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:25.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:25.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:25.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:25.340+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:25.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:25.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:25.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:25.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:25.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:25.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:25.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:25.405+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:26.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:25.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:26.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:25.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:26.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:25.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:26.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:25.732+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:26.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:26.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:26.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:26.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:26.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:26.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:26.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:26.318+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:26.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:26.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:26.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:26.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:26.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:26.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:26.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:26.387+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:27.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:26.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:27.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:26.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:27.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:26.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:27.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:26.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:27.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:27.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:27.806 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:27.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:27.806 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:27.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:27.806 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:27.331+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:27.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:27.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:27.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:27.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:27.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:27.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:27.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:27.367+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:28.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:27.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:28.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:27.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:28.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:27.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:28.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:27.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:28.555 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:28.145+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12193 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:28.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:28.354+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:28.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:28.354+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:28.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:28.354+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:28.556 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:28.355+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:28.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:28.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:28.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:28.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:28.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:28.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:28.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:28.379+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:29.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:28.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:29.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:28.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:29.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:28.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:29.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:28.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:29.801 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:29.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:29.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:29.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:29.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:29.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:29.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:29.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:29.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:29.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:29.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:29.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:29.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:29.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:29.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:29.372+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:30.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:29.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:30.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:29.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:30.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:29.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:30.056 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:29.788+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:30.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:30.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:30.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:30.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:30.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:30.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:30.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:30.327+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:30.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:30.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:30.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:30.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:30.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:30.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:30.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:30.398+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:31.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:30.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:31.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:30.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:31.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:30.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:31.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:30.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:31.733 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:31.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:31.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:31.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:31.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:31.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:31.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:31.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:31.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:31.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:31.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:31.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:31.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:31.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:31.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:31.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:32.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:31.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:32.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:31.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:32.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:31.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:32.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:31.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:32.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:32.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:32.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:32.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:32.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:32.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:32.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:32.403+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:32.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:32.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:32.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:32.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:32.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:32.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:32.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:32.403+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:33.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:32.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:33.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:32.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:33.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:32.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:33.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:32.726+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:33.425 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:33.147+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12287 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:33.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:33.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:33.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:33.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:33.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:33.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:33.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:33.440+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:33.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:33.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:33.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:33.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:33.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:33.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:33.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:33.411+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:34.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:33.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:34.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:33.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:34.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:33.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:34.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:33.764+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:34.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:34.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:34.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:34.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:34.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:34.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:34.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:34.373+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:34.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:34.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:34.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:34.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:34.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:34.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:34.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:34.476+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:35.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:34.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:35.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:34.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:35.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:34.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:35.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:34.756+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:35.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:35.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:35.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:35.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:35.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:35.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:35.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:35.390+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:35.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:35.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:35.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:35.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:35.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:35.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:35.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:35.503+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:35.998 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:35.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:35.998 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:35.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:35.999 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:35.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:35.999 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:35.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:36.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:36.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:36.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:36.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:36.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:36.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:36.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:36.397+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:36.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:36.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:36.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:36.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:36.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:36.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:36.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:36.498+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:37.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:36.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:37.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:36.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:37.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:36.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:37.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:36.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:37.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:37.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:37.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:37.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:37.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:37.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:37.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:37.358+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:37.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:37.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:37.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:37.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:37.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:37.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:37.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:37.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:38.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:37.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:38.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:37.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:38.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:37.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:38.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:37.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:38.395 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:38.149+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12379 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:38.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:38.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:38.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:38.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:38.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:38.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:38.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:38.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:38.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:38.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:38.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:38.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:38.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:38.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:38.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:38.382+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:39.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:38.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:39.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:38.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:39.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:38.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:39.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:38.755+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:39.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:39.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:39.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:39.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:39.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:39.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:39.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:39.371+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:39.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:39.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:39.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:39.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:39.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:39.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:39.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:39.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:40.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:39.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:40.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:39.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:40.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:39.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:40.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:39.732+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:40.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:40.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:40.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:40.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:40.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:40.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:40.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:40.409+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:40.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:40.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:40.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:40.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:40.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:40.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:40.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:40.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:41.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:40.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:41.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:40.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:41.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:40.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:41.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:40.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:41.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:41.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:41.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:41.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:41.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:41.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:41.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:41.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:41.719 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:41.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:41.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:41.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:41.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:41.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:41.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:41.402+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:42.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:41.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:42.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:41.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:42.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:41.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:42.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:41.705+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:42.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:42.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:42.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:42.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:42.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:42.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:42.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:42.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:42.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:42.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:42.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:42.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:42.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:42.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:42.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:42.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:43.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:42.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:43.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:42.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:43.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:42.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:43.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:42.660+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:43.430 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:43.151+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12472 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:43.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:43.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:43.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:43.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:43.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:43.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:43.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:43.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:43.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:43.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:43.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:43.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:43.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:43.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:43.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:43.417+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:44.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:43.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:44.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:43.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:44.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:43.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:44.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:43.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:44.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:44.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:44.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:44.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:44.599 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:44.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:44.599 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:44.452+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:45.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:44.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:45.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:44.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:45.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:44.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:45.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:44.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:45.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:44.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:45.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:44.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:45.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:44.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:45.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:44.584+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:45.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:45.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:45.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:45.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:45.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:45.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:45.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:45.604+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:45.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:45.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:45.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:45.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:45.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:45.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:45.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:45.456+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:46.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:45.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:46.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:45.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:46.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:45.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:46.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:45.713+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:46.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:46.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:46.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:46.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:46.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:46.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:46.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:46.593+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:46.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:46.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:46.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:46.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:46.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:46.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:46.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:46.483+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:47.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:46.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:47.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:46.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:47.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:46.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:47.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:46.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:47.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:47.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:47.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:47.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:47.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:47.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:47.555 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:47.477+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:48.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:47.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:48.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:47.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:48.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:47.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:48.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:47.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:48.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:47.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:48.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:47.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:48.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:47.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:48.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:47.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:48.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:48.154+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12566 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:48.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:48.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:48.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:48.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:48.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:48.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:48.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:48.528+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:48.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:48.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:48.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:48.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:48.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:48.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:48.599+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:49.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:48.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:49.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:48.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:49.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:48.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:49.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:48.781+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:49.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:49.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:49.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:49.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:49.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:49.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:49.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:49.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:49.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:49.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:49.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:49.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:49.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:49.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:49.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:49.488+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:50.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:49.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:50.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:49.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:50.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:49.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:50.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:49.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:50.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:50.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:50.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:50.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:50.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:50.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:50.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:50.491+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:50.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:50.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:50.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:50.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:50.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:50.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:50.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:50.510+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:51.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:50.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:51.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:50.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:51.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:50.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:51.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:50.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:51.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:51.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:51.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:51.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:51.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:51.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:51.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:51.518+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:51.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:51.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:51.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:51.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:51.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:51.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:51.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:51.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:52.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:51.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:52.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:51.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:52.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:51.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:52.306 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:51.818+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:52.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:52.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:52.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:52.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:52.807 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:52.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:52.807 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:52.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:52.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:52.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:52.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:52.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:52.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:52.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:52.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:52.568+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:53.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:52.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:53.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:52.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:53.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:52.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:53.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:52.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:53.497 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:53.156+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12660 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:53.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:53.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:53.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:53.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:53.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:53.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:53.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:53.576+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:53.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:53.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:53.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:53.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:53.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:53.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:53.806 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:53.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:54.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:53.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:54.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:53.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:54.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:53.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:54.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:53.844+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:54.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:54.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:54.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:54.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:54.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:54.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:54.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:54.585+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:54.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:54.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:54.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:54.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:54.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:54.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:54.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:54.476+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:55.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:54.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:55.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:54.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:55.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:54.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:55.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:54.858+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:55.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:55.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:55.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:55.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:55.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:55.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:55.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:55.605+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:55.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:55.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:55.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:55.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:55.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:55.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:55.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:55.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:56.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:55.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:56.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:55.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:56.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:55.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:56.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:55.836+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:56.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:56.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:56.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:56.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:56.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:56.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:56.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:56.578+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:56.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:56.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:56.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:56.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:56.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:56.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:56.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:56.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:57.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:56.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:57.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:56.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:57.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:56.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:57.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:56.868+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:57.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:57.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:57.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:57.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:57.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:57.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:57.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:57.558+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:57.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:57.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:57.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:57.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:57.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:57.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:57.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:57.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:58.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:57.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:58.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:57.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:58.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:57.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:58.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:57.836+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:58.554 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:18:58.158+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12753 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:18:58.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:58.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:58.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:58.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:58.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:58.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:58.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:58.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:58.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:58.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:58.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:58.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:58.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:58.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:58.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:58.550+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:18:59.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:58.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:59.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:58.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:59.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:58.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:18:59.305 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:58.803+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:59.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:59.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:59.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:59.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:59.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:59.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:18:59.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:18:59.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:18:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:59.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:59.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:59.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:59.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:18:59.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:18:59.597+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:00.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:59.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:00.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:59.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:00.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:59.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:00.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:18:59.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:00.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:00.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:00.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:00.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:00.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:00.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:00.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:00.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:00.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:00.581+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:00.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:00.581+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:00.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:00.581+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:00.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:00.582+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:01.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:00.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:01.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:00.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:01.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:00.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:01.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:00.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:01.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:01.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:01.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:01.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:01.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:01.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:01.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:01.762+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:01.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:01.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:01.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:01.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:01.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:01.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:01.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:01.572+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:01.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:01.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:01.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:01.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:01.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:01.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:01.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:01.514+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:02.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:02.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:02.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:02.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:02.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:02.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:02.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:02.607+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:02.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:02.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:02.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:02.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:02.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:02.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:02.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:02.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:02.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:02.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:02.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:02.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:02.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:02.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:02.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:02.748+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:03.454 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:03.160+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12846 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:04.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:03.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:04.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:03.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:04.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:03.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:04.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:03.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:04.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:03.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:04.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:03.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:04.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:03.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:04.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:03.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:04.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:03.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:04.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:03.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:04.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:03.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:04.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:03.559+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:05.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:04.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:05.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:04.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:05.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:04.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:05.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:04.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:05.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:04.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:05.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:04.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:05.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:04.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:05.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:04.650+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:05.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:04.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:05.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:04.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:05.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:04.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:05.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:04.592+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:06.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:05.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:06.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:05.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:06.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:05.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:06.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:05.689+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:06.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:05.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:06.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:05.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:06.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:05.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:06.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:05.640+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:06.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:05.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:06.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:05.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:06.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:05.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:06.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:05.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:07.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:06.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:07.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:06.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:07.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:06.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:07.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:06.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:07.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:06.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:07.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:06.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:07.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:06.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:07.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:06.674+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:07.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:07.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:07.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:07.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:06.682+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:08.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:07.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:08.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:07.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:08.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:07.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:08.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:07.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:08.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:07.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:08.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:07.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:08.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:07.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:08.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:07.639+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:08.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:07.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:08.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:07.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:08.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:07.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:08.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:07.699+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:08.554 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:08.162+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 12940 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:09.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:08.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:09.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:08.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:09.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:08.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:09.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:08.647+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:09.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:08.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:09.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:08.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:09.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:08.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:09.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:08.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:09.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:08.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:09.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:08.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:09.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:08.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:09.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:08.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:09.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:09.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:09.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:09.689+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:09.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:09.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:09.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:09.615+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:10.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:09.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:10.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:09.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:10.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:09.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:10.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:09.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:11.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:10.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:11.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:10.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:11.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:10.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:11.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:10.741+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:11.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:10.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:11.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:10.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:11.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:10.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:11.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:10.643+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:11.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:10.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:11.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:10.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:11.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:10.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:11.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:10.664+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:12.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:11.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:12.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:11.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:12.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:11.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:12.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:11.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:12.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:11.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:12.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:11.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:12.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:11.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:12.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:11.669+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:12.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:11.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:12.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:11.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:12.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:11.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:12.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:11.687+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:13.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:12.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:13.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:12.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:13.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:12.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:13.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:12.655+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:13.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:12.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:13.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:12.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:13.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:12.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:13.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:12.713+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:13.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:12.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:13.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:12.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:13.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:12.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:13.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:12.743+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:13.554 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:13.164+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13034 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:14.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:13.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:14.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:13.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:14.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:13.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:14.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:13.629+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:14.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:13.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:14.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:13.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:14.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:13.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:14.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:13.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:14.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:13.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:14.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:13.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:14.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:13.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:14.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:13.739+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:15.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:14.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:15.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:14.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:15.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:14.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:15.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:14.754+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:15.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:14.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:15.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:14.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:15.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:14.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:15.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:14.605+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:15.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:14.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:15.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:14.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:15.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:14.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:15.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:14.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:15.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:15.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:15.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:15.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:15.614+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:15.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:15.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:15.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:15.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:15.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:15.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:15.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:15.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:15.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:15.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:15.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:17.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:16.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:17.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:16.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:17.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:16.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:17.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:16.584+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:17.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:16.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:17.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:16.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:17.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:16.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:17.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:16.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:17.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:16.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:17.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:16.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:17.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:16.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:17.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:16.733+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:18.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:17.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:18.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:17.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:18.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:17.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:18.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:17.610+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:18.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:17.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:18.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:17.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:18.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:17.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:18.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:17.724+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:18.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:17.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:18.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:17.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:18.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:17.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:18.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:17.750+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:18.554 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:18.167+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13127 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:19.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:18.602+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:19.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:18.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:19.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:18.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:19.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:18.603+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:19.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:18.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:19.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:18.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:19.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:18.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:19.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:18.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:19.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:18.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:19.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:18.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:19.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:18.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:19.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:18.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:19.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:19.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:19.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:19.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:19.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:19.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:19.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:19.687+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:19.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:19.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:19.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:19.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:19.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:19.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:19.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:19.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:19.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:19.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:19.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:19.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:19.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:19.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:19.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:19.637+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:21.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:20.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:21.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:20.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:21.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:20.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:21.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:20.679+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:21.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:20.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:21.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:20.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:21.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:20.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:21.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:20.685+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:21.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:20.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:21.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:20.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:21.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:20.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:21.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:20.588+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:22.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:21.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:22.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:21.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:22.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:21.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:22.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:21.726+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:22.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:21.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:22.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:21.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:22.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:21.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:22.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:21.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:22.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:21.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:22.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:21.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:22.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:21.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:22.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:21.608+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:23.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:22.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:23.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:22.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:23.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:22.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:23.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:22.688+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:23.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:22.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:23.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:22.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:23.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:22.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:23.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:22.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:23.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:22.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:23.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:22.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:23.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:22.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:23.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:22.654+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:23.553 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:23.169+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13220 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:24.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:23.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:24.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:23.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:24.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:23.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:24.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:23.759+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:24.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:23.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:24.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:23.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:24.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:23.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:24.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:23.688+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:24.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:23.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:24.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:23.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:24.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:23.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:24.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:23.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:25.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:24.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:25.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:24.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:25.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:24.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:25.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:24.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:25.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:24.701+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:25.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:24.701+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:25.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:24.701+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:25.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:24.702+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:25.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:24.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:25.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:24.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:25.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:24.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:25.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:24.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:26.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:25.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:26.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:25.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:26.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:25.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:26.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:25.762+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:26.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:25.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:26.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:25.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:26.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:25.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:26.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:25.661+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:26.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:25.645+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:26.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:25.645+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:26.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:25.645+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:26.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:25.645+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:27.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:26.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:27.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:26.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:27.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:26.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:27.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:26.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:27.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:26.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:27.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:26.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:27.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:26.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:27.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:26.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:27.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:26.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:27.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:26.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:27.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:26.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:27.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:26.629+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:28.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:27.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:28.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:27.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:28.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:27.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:28.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:27.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:28.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:27.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:28.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:27.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:28.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:27.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:28.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:27.596+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:28.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:27.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:28.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:27.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:28.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:27.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:28.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:27.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:28.554 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:28.171+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13314 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:29.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:28.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:29.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:28.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:29.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:28.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:29.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:28.599+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:29.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:28.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:29.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:28.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:29.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:28.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:29.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:28.601+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:29.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:28.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:29.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:28.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:29.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:28.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:29.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:28.840+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:29.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:29.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:29.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:29.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:29.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:29.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:29.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:29.630+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:29.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:29.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:29.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:29.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:29.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:29.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:29.805 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:29.644+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:30.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:29.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:30.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:29.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:30.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:29.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:30.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:29.802+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:30.799 INFO:teuthology.orchestra.run.smithi077.stderr:2024-02-19T19:19:30.785+0000 7f05d45c3700 0 monclient(hunting): authenticate timed out after 300 2024-02-19T19:19:30.799 INFO:teuthology.orchestra.run.smithi077.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-02-19T19:19:31.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:30.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:30.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:30.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:30.679+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:30.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:30.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:30.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:30.827+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:30.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:31.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:30.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:31.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:30.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:31.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:30.629+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-19T19:19:31.501 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi077 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-19T19:19:31.694 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fd340e7905d94c4bbfe20140ba0ea85f Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi077 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-02-19T19:19:31.697 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-19T19:19:31.709 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-19T19:19:31.709 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T19:19:31.709 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:19:31.734 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T19:19:31.734 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:19:31.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T19:19:31.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:31.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:31.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:31.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:31.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:31.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:31.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:31.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:31.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:31.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:31.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:31.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:31.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:31.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:31.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:31.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:32.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:31.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:32.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:31.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:32.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:31.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:32.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:31.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:33.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:32.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:33.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:32.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:33.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:32.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:33.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:32.827+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:33.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:32.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:33.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:32.718+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:33.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:32.718+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:33.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:32.718+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:33.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:32.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:33.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:32.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:33.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:32.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:33.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:32.691+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:33.553 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:33.173+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13409 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:34.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:33.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:34.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:33.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:34.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:33.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:34.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:33.723+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:34.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:33.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:34.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:33.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:34.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:33.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:34.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:33.817+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:34.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:33.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:34.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:33.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:34.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:33.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:34.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:33.672+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:35.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:34.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:35.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:34.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:35.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:34.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:35.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:34.752+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:35.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:34.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:35.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:34.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:35.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:34.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:35.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:34.805+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:35.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:34.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:35.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:34.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:35.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:34.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:35.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:34.718+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:36.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:35.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:36.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:35.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:36.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:35.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:36.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:35.762+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:36.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:35.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:36.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:35.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:36.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:35.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:36.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:35.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:36.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:35.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:36.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:35.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:36.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:35.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:36.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:35.692+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:37.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:36.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:37.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:36.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:37.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:36.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:37.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:36.735+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:37.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:36.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:37.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:36.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:37.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:36.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:37.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:36.799+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:37.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:36.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:37.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:36.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:37.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:36.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:37.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:36.673+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:37.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:37.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:37.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:37.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:37.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:37.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:37.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:37.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:38.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:37.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:38.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:37.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:38.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:37.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:38.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:37.804+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:38.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:37.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:38.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:37.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:38.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:37.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:38.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:37.726+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:38.313 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:38.175+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13502 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:39.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:38.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:39.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:38.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:39.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:38.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:39.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:38.745+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:39.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:38.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:39.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:38.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:39.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:38.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:39.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:38.619+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:39.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:38.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:39.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:38.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:39.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:38.800+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:39.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:38.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:40.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:39.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:40.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:39.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:40.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:39.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:40.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:39.745+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:40.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:39.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:40.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:39.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:40.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:39.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:40.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:39.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:40.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:39.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:40.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:39.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:40.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:39.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:40.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:39.603+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:40.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:40.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:40.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:40.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:40.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:40.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:40.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:40.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:40.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:40.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:40.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:40.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:40.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:40.589+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:41.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:40.747+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:41.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:40.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:41.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:40.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:41.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:40.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:41.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:40.777+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:41.743 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:41.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:41.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:41.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:41.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:41.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:41.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:41.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:42.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:41.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:42.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:41.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:42.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:41.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:42.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:41.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:42.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:41.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:42.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:41.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:42.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:41.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:42.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:41.784+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:43.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:42.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:43.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:42.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:43.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:42.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:43.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:42.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:43.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:42.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:43.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:42.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:43.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:42.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:43.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:42.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:43.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:42.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:43.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:42.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:43.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:42.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:43.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:42.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:43.553 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:43.177+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13596 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:44.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:43.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:44.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:43.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:44.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:43.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:44.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:43.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:44.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:43.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:44.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:43.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:44.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:43.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:44.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:43.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:44.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:43.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:44.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:43.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:44.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:43.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:44.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:43.694+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:45.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:44.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:45.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:44.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:45.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:44.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:45.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:44.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:45.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:44.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:45.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:44.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:45.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:44.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:45.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:44.711+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:45.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:44.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:45.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:44.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:45.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:44.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:45.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:44.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:46.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:45.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:46.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:45.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:46.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:45.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:46.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:45.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:46.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:45.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:46.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:45.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:46.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:45.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:46.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:45.722+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:46.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:45.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:46.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:45.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:46.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:45.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:46.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:45.592+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:47.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:46.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:47.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:46.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:47.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:46.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:47.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:46.688+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:47.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:46.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:47.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:46.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:47.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:46.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:47.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:46.747+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:47.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:46.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:47.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:46.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:47.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:46.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:47.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:46.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:48.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:47.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:48.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:47.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:48.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:47.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:48.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:47.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:48.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:47.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:48.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:47.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:48.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:47.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:48.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:47.747+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:48.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:47.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:48.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:47.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:48.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:47.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:48.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:47.605+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:48.553 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:48.179+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13690 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:49.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:48.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:49.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:48.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:49.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:48.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:49.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:48.725+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:49.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:48.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:49.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:48.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:49.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:48.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:49.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:48.759+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:49.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:48.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:49.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:48.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:49.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:48.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:49.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:48.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:50.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:49.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:50.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:49.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:50.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:49.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:50.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:49.764+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:50.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:49.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:50.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:49.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:50.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:49.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:50.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:49.771+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:50.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:49.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:50.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:49.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:50.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:49.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:50.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:49.602+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:51.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:50.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:51.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:50.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:51.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:50.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:51.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:50.791+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:51.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:50.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:51.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:50.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:51.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:50.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:51.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:50.791+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:51.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:50.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:51.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:50.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:51.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:50.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:51.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:50.623+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:52.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:51.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:52.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:51.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:52.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:51.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:52.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:51.770+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:52.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:51.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:52.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:51.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:52.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:51.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:52.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:51.782+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:52.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:51.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:52.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:51.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:52.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:51.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:52.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:51.656+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:53.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:52.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:53.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:52.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:53.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:52.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:53.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:52.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:53.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:52.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:53.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:52.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:53.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:52.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:53.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:52.784+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:53.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:52.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:53.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:52.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:53.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:52.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:53.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:52.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:53.553 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:53.181+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13785 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:54.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:53.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:54.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:53.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:54.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:53.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:54.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:53.635+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:54.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:53.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:54.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:53.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:54.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:53.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:54.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:53.722+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:54.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:53.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:54.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:53.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:54.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:53.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:54.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:53.736+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:55.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:54.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:55.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:54.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:55.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:54.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:55.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:54.627+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:55.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:54.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:55.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:54.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:55.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:54.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:55.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:54.692+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:55.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:54.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:55.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:54.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:55.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:54.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:55.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:54.772+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:56.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:55.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:56.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:55.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:56.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:55.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:56.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:55.635+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:56.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:55.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:56.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:55.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:56.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:55.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:56.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:55.660+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:56.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:55.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:56.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:55.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:56.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:55.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:56.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:55.808+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:57.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:56.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:57.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:56.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:57.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:56.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:57.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:56.601+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:57.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:56.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:57.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:56.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:57.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:56.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:57.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:56.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:57.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:56.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:57.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:56.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:57.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:56.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:57.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:56.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:58.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:57.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:58.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:57.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:58.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:57.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:58.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:57.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:58.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:57.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:58.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:57.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:58.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:57.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:58.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:57.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:58.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:57.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:58.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:57.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:58.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:57.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:58.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:57.841+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:58.553 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:19:58.183+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13879 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:19:59.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:58.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:59.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:58.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:59.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:58.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:59.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:58.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:59.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:58.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:59.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:58.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:59.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:58.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:19:59.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:58.741+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:19:59.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:58.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:59.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:58.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:59.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:58.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:19:59.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:58.864+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:19:59.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:59.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:59.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:59.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:59.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:59.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:19:59.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:19:59.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:00.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:59.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:00.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:59.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:00.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:59.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:00.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:19:59.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:00.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:59.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:00.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:59.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:00.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:59.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:00.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:19:59.874+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:01.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:00.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:01.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:00.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:01.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:00.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:01.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:00.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:01.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:00.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:01.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:00.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:01.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:00.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:01.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:00.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:01.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:00.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:01.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:00.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:01.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:00.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:01.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:00.828+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:02.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:01.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:02.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:01.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:02.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:01.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:02.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:01.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:02.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:01.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:02.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:01.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:02.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:01.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:02.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:01.693+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:02.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:01.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:02.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:01.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:02.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:01.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:02.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:01.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:03.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:02.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:03.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:02.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:03.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:02.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:03.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:02.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:03.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:02.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:03.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:02.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:03.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:02.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:03.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:02.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:03.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:02.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:03.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:02.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:03.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:02.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:03.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:02.762+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:03.553 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:03.186+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 13973 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:03.882 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:03.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:03.883 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:03.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:03.883 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:03.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:03.883 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:03.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:03.883 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:03.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:03.883 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:03.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:03.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:03.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:03.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:03.717+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:04.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:03.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:04.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:03.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:04.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:03.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:04.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:03.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:05.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:04.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:05.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:04.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:05.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:04.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:05.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:04.856+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:05.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:04.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:05.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:04.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:05.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:04.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:05.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:04.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:05.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:04.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:05.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:04.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:05.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:04.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:05.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:04.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:06.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:05.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:06.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:05.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:06.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:05.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:06.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:05.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:06.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:05.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:06.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:05.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:06.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:05.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:06.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:05.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:06.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:05.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:06.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:05.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:06.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:05.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:06.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:05.687+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:07.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:06.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:07.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:06.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:07.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:06.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:07.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:06.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:07.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:06.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:07.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:06.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:07.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:06.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:07.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:06.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:07.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:06.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:07.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:06.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:07.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:06.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:07.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:06.665+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:08.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:07.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:08.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:07.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:08.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:07.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:08.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:07.940+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:08.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:07.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:08.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:07.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:08.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:07.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:08.054 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:07.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:08.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:07.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:08.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:07.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:08.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:07.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:08.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:07.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:08.552 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:08.187+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14068 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:08.820 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:08.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:08.820 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:08.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:08.820 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:08.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:08.820 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:08.731+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:09.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:08.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:09.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:08.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:09.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:08.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:09.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:08.808+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:09.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:08.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:09.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:08.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:09.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:08.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:09.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:08.968+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:10.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:09.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:10.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:09.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:10.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:09.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:10.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:09.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:10.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:09.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:10.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:09.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:10.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:09.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:10.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:09.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:10.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:09.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:10.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:09.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:10.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:09.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:10.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:09.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:11.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:10.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:11.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:10.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:11.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:10.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:11.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:10.857+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:11.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:11.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:11.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:11.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:11.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:11.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:11.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:11.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:11.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:10.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:11.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:10.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:11.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:10.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:11.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:10.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:12.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:11.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:12.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:11.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:12.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:11.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:12.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:11.827+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:12.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:11.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:12.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:11.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:12.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:11.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:12.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:11.750+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:12.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:12.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:12.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:12.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:12.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:12.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:12.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:12.058+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:13.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:12.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:13.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:12.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:13.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:12.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:13.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:12.867+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:13.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:12.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:13.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:12.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:13.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:12.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:13.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:12.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:13.302 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:13.190+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14163 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:13.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:13.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:13.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:13.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:13.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:13.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:13.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:13.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:13.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:13.748+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:13.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:13.748+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:13.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:13.748+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:13.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:13.748+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:14.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:13.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:14.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:13.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:14.301 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:13.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:14.301 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:13.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:14.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:14.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:14.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:14.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:14.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:14.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:14.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:14.052+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:15.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:14.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:15.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:14.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:15.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:14.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:15.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:14.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:15.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:14.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:15.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:14.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:15.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:14.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:15.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:14.770+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:15.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:15.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:15.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:15.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:15.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:15.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:15.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:15.052+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:16.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:15.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:16.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:15.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:16.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:15.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:16.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:15.796+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:16.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:15.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:16.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:15.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:16.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:15.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:16.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:15.824+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:16.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:16.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:16.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:16.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:16.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:16.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:16.305 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:16.023+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:17.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:16.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:17.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:16.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:17.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:16.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:17.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:16.823+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:17.303 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:16.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:17.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:16.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:17.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:16.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:17.304 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:16.803+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:17.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:17.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:17.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:17.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:17.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:17.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:17.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:17.031+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:18.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:17.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:18.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:17.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:18.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:17.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:18.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:17.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:18.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:18.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:18.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:18.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:18.053 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:18.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:18.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:18.011+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:18.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:17.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:18.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:17.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:18.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:17.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:18.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:17.851+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:18.552 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:18.193+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14256 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:19.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:18.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:19.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:18.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:19.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:18.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:19.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:18.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:19.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:18.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:19.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:18.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:19.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:18.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:19.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:18.811+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:19.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:19.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:19.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:19.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:19.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:19.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:19.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:19.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:19.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:19.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:19.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:19.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:19.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:19.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:19.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:19.701+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:20.163 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:19.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:20.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:19.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:20.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:19.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:20.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:19.836+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:20.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:20.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:20.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:20.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:20.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:20.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:20.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:20.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:21.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:20.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:21.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:20.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:21.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:20.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:21.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:20.797+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:21.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:20.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:21.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:20.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:21.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:20.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:21.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:20.710+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:21.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:21.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:21.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:21.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:21.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:21.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:21.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:21.057+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:22.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:21.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:22.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:21.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:22.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:21.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:22.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:21.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:22.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:21.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:22.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:21.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:22.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:21.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:22.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:21.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:22.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:22.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:22.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:22.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:22.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:22.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:22.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:22.071+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:23.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:22.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:23.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:22.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:23.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:22.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:23.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:22.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:23.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:22.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:23.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:22.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:23.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:22.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:23.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:22.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:23.552 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:23.195+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14351 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:23.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:23.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:23.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:23.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:23.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:23.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:23.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:23.105+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:24.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:23.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:24.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:23.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:24.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:23.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:24.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:23.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:24.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:23.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:24.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:23.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:24.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:23.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:24.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:23.712+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:24.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:24.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:24.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:24.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:24.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:24.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:24.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:24.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:25.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:24.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:25.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:24.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:25.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:24.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:25.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:24.678+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:25.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:24.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:25.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:24.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:25.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:24.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:25.054 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:24.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:25.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:25.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:25.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:25.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:25.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:25.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:25.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:25.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:26.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:25.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:26.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:25.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:26.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:25.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:26.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:25.670+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:26.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:25.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:26.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:25.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:26.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:25.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:26.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:25.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:26.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:26.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:26.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:26.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:26.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:26.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:26.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:26.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:27.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:26.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:27.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:26.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:27.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:26.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:27.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:26.630+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:27.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:26.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:27.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:26.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:27.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:26.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:27.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:26.793+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:27.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:27.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:27.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:27.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:27.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:27.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:27.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:27.067+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:28.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:27.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:28.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:27.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:28.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:27.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:28.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:27.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:28.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:27.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:28.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:27.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:28.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:27.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:28.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:27.801+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:28.552 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:28.197+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14444 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:28.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:28.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:28.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:28.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:28.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:28.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:28.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:28.096+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:29.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:28.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:29.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:28.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:29.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:28.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:29.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:28.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:29.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:28.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:29.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:28.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:29.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:28.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:29.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:28.768+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:29.509 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:29.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:29.509 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:29.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:29.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:29.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:29.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:29.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:29.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:29.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:29.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:29.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:29.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:29.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:29.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:29.568+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:30.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:30.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:30.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:30.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:30.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:30.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:30.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:30.016+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:30.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:29.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:30.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:29.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:30.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:29.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:30.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:29.812+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:31.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:30.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:31.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:30.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:31.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:30.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:31.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:30.606+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:31.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:30.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:31.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:30.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:31.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:30.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:31.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:30.794+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:31.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:31.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:31.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:31.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:31.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:31.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:31.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:31.059+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:32.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:31.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:32.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:31.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:32.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:31.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:32.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:31.766+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:32.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:31.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:32.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:31.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:32.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:31.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:32.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:31.564+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:32.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:32.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:32.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:32.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:32.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:32.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:32.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:32.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:33.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:32.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:33.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:32.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:33.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:32.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:33.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:32.565+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:33.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:32.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:33.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:32.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:33.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:32.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:33.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:32.811+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:33.337 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:33.200+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14539 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:33.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:33.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:33.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:33.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:33.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:33.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:33.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:33.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:34.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:33.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:34.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:33.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:34.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:33.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:34.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:33.589+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:33.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:33.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:33.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:33.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:34.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:34.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:34.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:34.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:34.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:34.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:34.553 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:34.109+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:35.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:34.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:35.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:34.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:35.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:34.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:35.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:34.800+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:35.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:34.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:35.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:34.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:35.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:34.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:35.053 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:34.575+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:35.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:35.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:35.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:35.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:35.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:35.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:35.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:35.092+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:35.847 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:35.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:35.847 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:35.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:35.847 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:35.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:35.847 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:35.552+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:36.133 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:35.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:36.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:35.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:36.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:35.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:36.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:35.836+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:36.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:36.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:36.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:36.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:36.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:36.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:36.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:36.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:36.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:36.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:36.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:36.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:36.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:36.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:36.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:36.507+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:37.171 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:36.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:37.171 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:36.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:37.171 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:36.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:37.171 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:36.804+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:37.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:37.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:37.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:37.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:37.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:37.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:37.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:37.161+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:37.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:37.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:37.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:37.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:37.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:37.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:37.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:37.483+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:38.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:37.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:38.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:37.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:38.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:37.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:38.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:37.763+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:38.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:38.202+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14631 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:38.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:38.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:38.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:38.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:38.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:38.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:38.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:38.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:38.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:38.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:38.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:38.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:38.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:38.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:38.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:38.522+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:39.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:38.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:39.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:38.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:39.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:38.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:39.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:38.792+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:39.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:39.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:39.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:39.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:39.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:39.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:39.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:39.475+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:39.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:39.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:39.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:39.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:39.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:39.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:39.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:39.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:40.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:39.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:40.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:39.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:40.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:39.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:40.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:39.841+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:40.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:40.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:40.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:40.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:40.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:40.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:40.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:40.276+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:40.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:40.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:40.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:40.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:40.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:40.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:40.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:40.494+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:41.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:40.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:41.262 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:40.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:41.262 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:40.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:41.262 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:40.813+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:41.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:41.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:41.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:41.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:41.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:41.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:41.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:41.251+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:41.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:41.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:41.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:41.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:41.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:41.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:41.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:41.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:42.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:41.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:42.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:41.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:42.216 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:41.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:42.216 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:41.863+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:42.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:42.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:42.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:42.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:42.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:42.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:42.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:42.205+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:42.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:42.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:42.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:42.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:42.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:42.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:42.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:42.516+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:43.216 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:42.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:43.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:42.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:43.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:42.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:43.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:42.828+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:43.489 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:43.205+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14725 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:43.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:43.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:43.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:43.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:43.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:43.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:43.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:43.218+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:43.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:43.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:43.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:43.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:43.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:43.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:43.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:43.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:44.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:43.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:44.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:43.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:44.250 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:43.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:44.250 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:43.877+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:44.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:44.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:44.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:44.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:44.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:44.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:44.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:44.239+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:44.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:44.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:44.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:44.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:44.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:44.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:44.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:44.497+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:45.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:44.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:45.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:44.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:45.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:44.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:45.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:44.888+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:45.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:45.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:45.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:45.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:45.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:45.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:45.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:45.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:45.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:45.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:45.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:45.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:45.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:45.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:45.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:45.477+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:46.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:45.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:46.186 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:45.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:46.186 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:45.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:46.186 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:45.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:46.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:46.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:46.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:46.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:46.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:46.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:46.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:46.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:46.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:46.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:46.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:46.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:46.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:46.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:46.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:46.495+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:47.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:46.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:47.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:46.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:47.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:46.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:47.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:46.892+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:47.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:47.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:47.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:47.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:47.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:47.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:47.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:47.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:47.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:47.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:47.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:47.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:47.501+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:48.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:47.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:48.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:47.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:48.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:47.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:48.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:47.918+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:48.531 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:48.207+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14815 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:48.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:48.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:48.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:48.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:48.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:48.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:48.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:48.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:48.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:48.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:48.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:48.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:48.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:48.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:48.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:48.521+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:49.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:48.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:49.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:48.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:49.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:48.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:49.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:48.968+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:49.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:49.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:49.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:49.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:49.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:49.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:49.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:49.248+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:49.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:49.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:49.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:49.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:49.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:49.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:49.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:49.530+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:50.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:50.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:50.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:50.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:50.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:50.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:50.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:50.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:50.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:49.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:50.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:49.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:50.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:49.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:50.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:49.963+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:50.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:50.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:50.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:50.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:50.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:50.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:50.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:50.567+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:51.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:51.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:51.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:51.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:51.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:51.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:51.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:51.174+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:51.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:50.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:51.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:50.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:51.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:50.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:51.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:50.963+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:51.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:51.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:51.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:51.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:51.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:51.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:51.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:51.615+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:52.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:52.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:52.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:52.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:52.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:52.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:52.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:52.154+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:52.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:51.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:52.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:51.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:52.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:51.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:52.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:51.926+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:52.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:52.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:52.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:52.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:52.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:52.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:52.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:52.609+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:53.232 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:53.210+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14904 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:53.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:53.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:53.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:53.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:53.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:53.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:53.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:53.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:53.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:52.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:53.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:52.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:53.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:52.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:53.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:52.966+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:54.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:53.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:54.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:53.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:54.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:53.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:54.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:53.573+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:54.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:54.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:54.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:54.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:54.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:54.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:54.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:54.131+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:54.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:54.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:54.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:54.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:54.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:54.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:54.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:54.007+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:54.981 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:54.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:54.981 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:54.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:54.981 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:54.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:54.982 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:54.590+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:55.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:55.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:55.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:55.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:55.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:55.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:55.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:55.100+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:55.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:54.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:55.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:54.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:55.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:54.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:55.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:54.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:55.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:55.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:55.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:55.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:55.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:55.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:55.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:55.597+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:56.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:56.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:56.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:56.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:56.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:56.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:56.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:56.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:56.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:55.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:56.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:55.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:56.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:55.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:56.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:55.950+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:56.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:56.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:56.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:56.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:56.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:56.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:56.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:56.631+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:57.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:57.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:57.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:57.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:57.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:57.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:57.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:57.089+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:57.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:56.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:57.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:56.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:57.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:56.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:57.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:56.916+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:57.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:57.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:57.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:57.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:57.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:57.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:57.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:57.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:58.234 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:20:58.212+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 14990 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:20:58.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:57.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:58.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:57.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:58.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:57.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:58.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:57.964+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:58.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:58.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:58.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:58.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:58.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:58.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:58.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:58.060+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:58.971 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:58.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:58.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:58.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:58.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:58.637+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:58.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:58.639+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:59.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:59.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:59.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:59.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:59.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:59.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:20:59.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:20:59.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:20:59.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:58.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:59.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:58.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:59.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:58.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:20:59.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:58.961+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:20:59.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:59.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:59.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:59.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:59.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:59.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:20:59.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:20:59.593+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:00.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:00.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:00.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:00.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:00.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:00.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:00.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:00.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:00.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:59.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:00.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:59.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:00.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:59.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:00.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:20:59.920+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:00.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:00.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:00.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:00.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:00.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:00.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:00.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:00.575+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:01.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:00.892+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:01.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:00.892+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:01.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:00.892+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:01.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:00.893+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:01.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:01.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:01.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:01.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:01.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:01.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:01.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:01.167+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:01.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:01.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:01.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:01.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:01.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:01.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:01.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:01.527+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:02.166 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:01.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:02.166 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:01.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:02.166 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:01.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:02.167 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:01.849+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:02.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:02.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:02.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:02.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:02.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:02.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:02.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:02.156+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:02.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:02.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:02.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:02.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:02.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:02.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:02.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:02.489+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:03.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:02.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:03.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:02.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:03.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:02.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:03.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:02.899+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:03.451 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:03.215+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15078 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:03.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:03.149+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:03.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:03.149+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:03.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:03.149+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:03.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:03.150+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:03.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:03.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:03.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:03.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:03.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:03.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:03.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:03.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:04.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:04.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:04.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:04.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:04.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:04.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:04.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:04.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:04.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:03.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:04.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:03.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:04.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:03.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:04.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:03.918+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:04.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:04.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:04.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:04.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:04.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:04.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:04.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:04.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:05.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:05.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:05.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:05.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:05.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:05.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:05.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:05.129+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:05.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:04.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:05.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:04.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:05.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:04.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:05.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:04.937+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:05.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:05.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:05.470+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:06.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:06.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:06.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:06.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:06.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:06.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:06.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:06.170+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:06.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:05.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:06.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:05.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:06.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:05.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:06.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:05.960+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:06.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:06.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:06.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:06.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:06.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:06.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:06.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:06.429+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:07.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:06.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:07.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:06.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:07.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:06.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:07.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:06.932+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:07.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:07.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:07.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:07.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:07.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:07.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:07.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:07.134+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:07.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:07.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:07.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:07.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:07.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:07.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:07.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:07.433+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:08.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:07.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:08.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:07.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:08.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:07.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:08.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:07.960+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:08.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:08.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:08.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:08.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:08.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:08.165+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:08.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:08.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:08.478 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:08.217+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15163 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:08.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:08.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:08.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:08.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:08.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:08.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:08.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:08.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:09.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:09.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:09.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:09.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:09.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:09.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:09.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:09.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:09.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:08.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:09.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:08.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:09.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:08.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:09.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:08.941+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:09.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:09.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:09.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:09.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:09.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:09.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:09.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:09.515+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:10.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:10.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:10.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:10.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:10.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:10.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:10.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:10.158+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:10.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:09.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:10.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:09.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:10.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:09.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:10.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:09.939+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:10.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:10.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:10.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:10.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:10.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:10.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:10.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:10.486+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:11.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:11.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:11.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:11.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:11.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:11.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:11.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:11.138+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:11.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:10.894+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:11.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:10.894+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:11.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:10.894+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:11.303 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:10.895+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:11.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:11.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:11.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:11.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:11.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:11.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:11.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:11.489+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:12.137 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:11.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:12.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:11.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:12.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:11.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:12.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:11.860+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:12.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:12.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:12.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:12.128+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:12.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:12.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:12.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:12.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:12.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:12.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:12.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:12.531+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:13.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:13.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:13.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:13.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:13.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:13.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:13.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:13.095+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:13.231 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:12.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:13.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:12.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:13.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:12.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:13.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:12.859+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:13.547 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:13.220+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15250 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:13.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:13.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:13.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:13.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:13.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:13.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:13.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:13.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:14.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:14.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:14.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:14.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:14.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:14.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:14.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:14.088+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:14.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:13.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:14.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:13.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:14.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:13.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:14.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:13.850+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:14.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:14.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:14.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:14.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:14.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:14.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:14.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:14.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:15.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:15.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:15.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:15.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:15.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:15.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:15.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:15.091+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:15.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:14.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:15.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:14.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:15.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:14.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:15.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:14.852+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:15.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:15.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:15.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:15.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:15.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:15.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:15.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:15.546+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:16.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:15.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:16.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:15.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:16.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:15.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:16.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:15.854+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:16.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:16.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:16.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:16.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:16.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:16.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:16.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:16.118+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:16.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:16.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:16.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:16.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:16.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:16.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:16.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:16.592+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:17.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:16.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:17.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:16.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:17.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:16.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:17.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:16.874+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:17.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:17.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:17.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:17.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:17.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:17.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:17.303 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:17.070+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:17.857 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:17.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:17.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:17.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:17.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:17.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:17.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:17.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:18.233 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:17.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:18.233 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:17.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:18.233 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:17.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:18.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:17.848+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:18.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:18.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:18.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:18.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:18.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:18.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:18.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:18.065+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:18.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:18.222+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15336 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:18.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:18.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:18.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:18.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:18.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:18.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:18.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:18.510+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:19.077 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:18.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:19.077 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:18.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:19.077 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:18.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:19.078 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:18.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:19.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:19.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:19.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:19.069+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:19.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:19.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:19.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:19.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:19.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:19.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:19.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:19.497+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:20.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:19.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:20.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:19.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:20.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:19.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:20.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:19.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:20.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:20.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:20.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:20.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:20.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:20.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:20.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:20.092+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:20.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:20.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:20.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:20.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:20.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:20.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:20.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:20.507+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:21.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:21.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:21.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:21.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:21.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:21.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:21.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:21.050+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:21.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:20.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:21.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:20.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:21.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:20.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:21.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:20.823+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:21.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:21.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:21.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:21.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:21.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:21.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:21.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:21.470+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:22.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:21.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:22.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:21.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:22.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:21.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:22.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:21.812+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:22.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:22.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:22.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:22.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:22.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:22.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:22.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:22.097+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:22.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:22.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:22.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:22.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:22.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:22.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:22.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:22.496+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:23.071 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:22.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:23.071 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:22.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:23.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:22.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:23.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:22.810+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:23.361 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:23.225+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15422 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:23.361 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:23.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:23.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:23.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:23.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:23.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:23.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:23.061+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:23.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:23.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:23.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:23.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:23.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:23.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:23.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:23.520+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:24.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:24.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:24.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:24.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:24.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:24.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:24.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:24.079+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:24.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:23.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:24.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:23.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:24.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:23.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:24.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:23.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:24.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:24.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:24.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:24.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:24.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:24.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:24.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:24.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:25.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:24.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:25.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:24.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:25.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:24.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:25.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:24.835+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:25.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:25.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:25.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:25.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:25.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:25.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:25.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:25.116+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:25.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:25.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:25.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:25.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:25.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:25.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:25.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:25.490+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:26.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:25.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:26.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:25.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:26.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:25.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:26.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:25.808+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:26.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:26.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:26.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:26.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:26.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:26.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:26.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:26.102+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:26.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:26.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:26.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:26.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:26.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:26.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:26.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:26.536+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:27.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:26.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:27.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:26.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:27.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:26.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:27.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:26.825+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:27.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:27.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:27.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:27.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:27.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:27.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:27.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:27.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:28.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:28.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:28.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:28.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:27.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:28.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:27.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:28.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:27.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:28.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:27.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:28.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:27.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:28.551 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:28.227+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15507 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:28.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:28.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:28.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:28.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:28.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:28.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:28.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:28.136+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:28.831 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:28.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:28.831 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:28.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:28.831 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:28.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:28.832 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:28.552+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:29.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:28.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:29.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:28.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:29.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:28.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:29.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:28.821+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:29.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:29.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:29.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:29.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:29.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:29.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:29.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:29.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:29.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:29.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:29.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:29.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:29.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:29.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:29.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:29.575+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:30.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:29.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:30.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:29.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:30.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:29.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:30.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:29.804+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:30.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:30.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:30.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:30.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:30.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:30.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:30.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:30.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:31.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:30.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:31.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:30.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:31.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:30.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:31.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:30.800+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:31.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:30.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:31.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:30.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:31.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:30.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:31.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:30.559+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:31.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:31.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:31.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:31.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:31.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:31.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:31.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:31.118+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:31.838 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:31.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:31.839 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:31.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:31.839 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:31.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:31.839 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:31.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:31.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:31.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:31.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:31.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:31.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:31.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:31.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:31.567+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-19T19:21:31.841 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-02-19T19:21:31.841 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T19:21:31.841 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:21:31.862 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-02-19T19:21:31.862 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:21:31.922 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-19T19:21:32.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:32.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:32.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:32.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:32.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:32.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:32.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:32.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:33.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:32.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:33.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:32.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:33.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:32.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:33.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:32.565+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:33.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:32.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:33.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:32.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:33.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:32.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:33.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:32.803+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:33.551 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:33.230+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15595 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:33.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:33.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:33.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:33.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:33.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:33.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:33.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:33.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:33.836 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:33.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:33.836 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:33.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:33.836 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:33.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:33.836 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:33.575+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:34.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:33.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:34.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:33.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:34.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:33.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:34.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:33.826+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:34.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:34.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:34.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:34.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:34.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:34.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:34.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:34.122+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:35.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:34.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:35.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:34.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:35.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:34.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:35.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:34.554+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:35.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:34.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:35.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:34.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:35.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:34.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:35.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:34.778+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:35.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:35.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:35.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:35.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:35.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:35.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:35.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:35.164+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:36.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:35.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:36.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:35.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:36.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:35.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:36.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:35.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:36.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:35.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:36.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:35.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:36.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:35.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:36.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:35.806+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:36.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:36.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:36.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:36.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:36.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:36.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:36.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:36.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:36.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:36.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:36.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:36.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:36.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:36.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:36.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:36.523+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:37.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:36.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:37.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:36.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:37.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:36.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:37.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:36.780+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:37.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:37.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:37.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:37.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:37.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:37.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:37.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:37.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:37.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:37.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:37.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:37.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:37.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:37.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:37.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:37.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:38.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:37.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:38.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:37.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:38.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:37.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:38.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:37.774+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:38.367 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:38.232+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15680 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:38.368 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:38.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:38.368 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:38.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:38.368 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:38.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:38.368 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:38.110+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:38.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:38.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:38.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:38.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:38.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:38.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:38.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:38.526+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:39.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:38.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:39.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:38.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:39.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:38.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:39.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:38.800+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:39.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:39.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:39.509 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:39.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:39.509 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:39.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:39.509 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:39.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:39.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:39.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:39.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:39.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:39.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:39.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:39.559+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:40.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:39.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:40.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:39.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:40.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:39.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:40.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:39.806+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:40.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:40.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:40.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:40.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:40.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:40.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:40.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:40.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:41.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:40.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:41.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:40.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:41.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:40.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:41.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:40.582+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:41.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:40.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:41.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:40.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:41.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:40.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:41.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:40.818+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:41.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:41.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:41.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:41.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:41.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:41.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:41.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:41.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:42.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:41.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:42.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:41.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:42.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:41.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:42.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:41.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:42.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:41.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:42.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:41.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:42.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:41.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:42.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:41.815+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:42.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:42.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:42.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:42.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:42.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:42.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:42.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:42.146+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:43.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:42.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:43.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:42.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:43.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:42.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:43.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:42.645+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:43.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:42.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:43.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:42.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:43.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:42.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:43.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:42.849+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:43.372 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:43.235+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15767 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:43.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:43.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:43.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:43.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:43.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:43.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:43.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:43.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:44.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:43.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:44.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:43.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:44.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:43.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:44.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:43.630+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:44.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:43.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:44.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:43.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:44.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:43.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:44.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:43.872+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:44.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:44.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:44.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:44.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:44.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:44.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:44.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:44.107+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:44.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:44.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:44.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:44.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:44.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:44.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:44.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:44.603+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:45.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:45.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:45.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:45.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:45.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:45.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:45.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:45.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:45.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:44.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:45.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:44.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:45.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:44.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:45.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:44.894+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:45.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:45.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:45.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:45.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:45.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:45.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:45.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:45.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:46.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:46.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:46.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:46.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:46.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:46.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:46.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:46.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:46.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:45.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:46.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:45.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:46.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:45.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:46.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:45.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:46.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:46.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:46.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:46.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:46.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:46.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:46.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:46.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:47.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:47.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:47.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:47.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:47.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:47.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:47.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:47.119+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:47.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:46.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:47.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:46.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:47.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:46.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:47.302 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:46.905+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:48.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:47.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:48.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:47.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:48.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:47.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:48.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:47.896+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:48.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:47.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:48.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:47.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:48.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:47.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:48.052 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:47.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:48.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:48.238+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15852 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:48.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:48.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:48.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:48.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:48.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:48.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:48.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:48.084+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:49.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:48.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:49.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:48.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:49.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:48.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:49.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:48.629+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:49.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:48.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:49.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:48.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:49.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:48.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:49.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:48.876+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:49.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:49.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:49.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:49.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:49.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:49.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:49.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:49.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:49.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:49.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:49.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:49.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:49.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:49.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:49.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:49.616+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:50.168 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:50.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:50.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:50.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:50.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:50.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:50.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:50.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:50.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:49.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:50.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:49.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:50.170 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:49.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:50.170 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:49.879+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:50.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:50.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:50.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:50.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:50.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:50.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:50.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:50.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:51.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:50.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:51.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:50.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:51.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:50.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:51.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:50.850+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:51.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:51.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:51.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:51.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:51.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:51.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:51.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:51.102+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:52.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:51.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:52.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:51.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:52.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:51.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:52.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:51.592+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:52.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:51.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:52.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:51.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:52.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:51.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:52.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:51.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:52.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:52.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:52.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:52.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:52.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:52.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:52.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:52.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:53.050 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:52.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:53.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:52.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:53.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:52.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:53.051 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:52.584+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:53.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:52.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:53.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:52.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:53.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:52.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:53.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:52.823+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:53.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:53.240+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 15932 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:53.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:53.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:53.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:53.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:53.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:53.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:53.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:53.187+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:53.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:53.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:53.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:53.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:53.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:53.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:53.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:53.541+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:54.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:53.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:54.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:53.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:54.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:53.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:54.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:53.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:54.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:54.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:54.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:54.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:54.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:54.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:54.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:54.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:54.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:54.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:54.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:54.492+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:55.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:54.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:55.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:54.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:55.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:54.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:55.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:54.772+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:55.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:55.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:55.535 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:55.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:55.535 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:55.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:55.535 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:55.189+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:55.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:55.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:55.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:55.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:55.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:55.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:55.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:55.525+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:56.179 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:55.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:56.180 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:55.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:56.180 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:55.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:56.180 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:55.792+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:56.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:56.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:56.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:56.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:56.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:56.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:56.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:56.170+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:56.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:56.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:56.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:56.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:56.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:56.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:56.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:56.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:57.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:56.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:57.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:56.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:57.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:56.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:57.197 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:56.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:57.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:57.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:57.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:57.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:57.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:57.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:57.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:57.187+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:57.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:57.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:57.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:57.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:57.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:57.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:57.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:57.533+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:58.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:57.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:58.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:57.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:58.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:57.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:58.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:57.865+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:58.539 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:21:58.243+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16017 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:21:58.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:58.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:58.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:58.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:58.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:58.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:58.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:58.237+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:58.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:58.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:58.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:58.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:58.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:58.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:58.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:58.530+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:59.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:58.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:59.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:58.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:59.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:58.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:21:59.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:58.834+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:21:59.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:59.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:59.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:59.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:59.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:59.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:21:59.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:21:59.220+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:21:59.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:59.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:59.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:59.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:59.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:59.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:21:59.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:21:59.489+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:00.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:59.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:00.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:59.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:00.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:59.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:00.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:21:59.843+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:00.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:00.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:00.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:00.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:00.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:00.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:00.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:00.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:00.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:00.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:00.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:00.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:00.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:00.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:00.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:00.474+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:01.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:00.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:01.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:00.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:01.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:00.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:01.178 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:00.887+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:01.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:01.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:01.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:01.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:01.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:01.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:01.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:01.168+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:01.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:01.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:01.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:01.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:01.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:01.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:01.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:01.491+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:02.141 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:01.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:02.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:01.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:02.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:01.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:02.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:01.909+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:02.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:02.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:02.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:02.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:02.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:02.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:02.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:02.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:02.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:02.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:02.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:02.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:02.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:02.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:02.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:02.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:03.184 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:02.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:03.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:02.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:03.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:02.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:03.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:02.908+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:03.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:03.245+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16105 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:03.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:03.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:03.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:03.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:03.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:03.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:03.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:03.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:03.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:03.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:03.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:03.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:03.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:03.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:03.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:03.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:04.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:03.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:04.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:03.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:04.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:03.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:04.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:03.925+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:04.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:04.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:04.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:04.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:04.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:04.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:04.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:04.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:04.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:04.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:04.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:04.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:04.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:04.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:04.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:04.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:05.200 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:04.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:05.200 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:04.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:05.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:04.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:05.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:04.932+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:05.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:05.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:05.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:05.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:05.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:05.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:05.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:05.191+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:05.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:05.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:05.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:05.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:05.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:06.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:05.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:06.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:05.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:06.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:05.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:06.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:05.946+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:06.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:06.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:06.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:06.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:06.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:06.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:06.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:06.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:06.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:06.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:06.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:06.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:06.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:06.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:06.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:06.473+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:07.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:06.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:07.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:06.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:07.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:06.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:07.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:06.938+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:07.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:07.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:07.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:07.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:07.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:07.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:07.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:07.189+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:07.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:07.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:07.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:07.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:07.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:07.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:07.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:07.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:08.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:07.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:08.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:07.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:08.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:07.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:08.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:07.984+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:08.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:08.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:08.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:08.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:08.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:08.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:08.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:08.176+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:08.550 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:08.248+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16191 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:08.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:08.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:08.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:08.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:08.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:08.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:08.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:08.480+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:09.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:09.030+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:09.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:09.030+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:09.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:09.030+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:09.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:09.030+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:09.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:09.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:09.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:09.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:09.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:09.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:09.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:09.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:09.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:09.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:09.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:09.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:09.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:09.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:09.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:09.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:10.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:10.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:10.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:10.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:10.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:10.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:10.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:10.032+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:10.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:10.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:10.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:10.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:10.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:10.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:10.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:10.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:10.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:10.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:10.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:10.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:10.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:10.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:10.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:10.497+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:11.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:11.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:11.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:11.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:11.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:11.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:11.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:11.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:11.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:10.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:11.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:10.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:11.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:10.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:11.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:10.998+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:11.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:11.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:11.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:11.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:11.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:11.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:11.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:11.462+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:12.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:12.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:12.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:12.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:12.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:12.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:12.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:12.031+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:12.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:12.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:12.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:12.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:12.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:12.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:12.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:12.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:12.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:12.449+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:12.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:12.449+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:12.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:12.449+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:12.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:12.450+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:13.385 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:13.250+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16280 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:13.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:13.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:13.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:13.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:13.386 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:13.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:13.386 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:13.057+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:13.386 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:13.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:13.386 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:13.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:13.386 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:13.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:13.386 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:13.106+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:13.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:13.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:13.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:13.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:13.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:13.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:13.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:13.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:14.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:14.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:14.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:14.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:14.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:14.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:14.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:14.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:14.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:14.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:14.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:14.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:14.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:14.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:14.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:14.082+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:14.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:14.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:14.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:14.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:14.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:14.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:14.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:14.466+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:15.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:15.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:15.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:15.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:15.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:15.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:15.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:15.202+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:15.440 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:15.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:15.440 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:15.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:15.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:15.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:15.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:15.097+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:15.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:15.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:15.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:15.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:15.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:15.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:15.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:15.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:16.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:16.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:16.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:16.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:16.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:16.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:16.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:16.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:16.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:16.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:16.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:16.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:16.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:16.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:16.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:16.138+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:16.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:16.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:16.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:16.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:16.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:16.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:16.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:16.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:17.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:17.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:17.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:17.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:17.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:17.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:17.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:17.202+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:17.467 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:17.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:17.467 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:17.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:17.468 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:17.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:17.468 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:17.116+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:17.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:17.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:17.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:17.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:17.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:17.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:17.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:17.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:18.494 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:18.253+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16366 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:18.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:18.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:18.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:18.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:18.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:18.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:18.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:18.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:18.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:18.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:18.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:18.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:18.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:18.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:18.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:18.147+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:18.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:18.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:18.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:18.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:18.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:18.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:18.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:18.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:19.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:19.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:19.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:19.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:19.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:19.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:19.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:19.127+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:19.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:19.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:19.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:19.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:19.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:19.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:19.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:19.105+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:19.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:19.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:19.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:19.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:19.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:19.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:19.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:19.530+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:20.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:20.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:20.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:20.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:20.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:20.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:20.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:20.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:20.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:20.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:20.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:20.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:20.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:20.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:20.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:20.112+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:20.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:20.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:20.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:20.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:20.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:20.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:20.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:20.530+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:21.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:21.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:21.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:21.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:21.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:21.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:21.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:21.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:21.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:21.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:21.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:21.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:21.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:21.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:21.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:21.130+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:21.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:21.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:21.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:21.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:21.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:21.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:21.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:21.512+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:22.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:22.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:22.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:22.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:22.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:22.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:22.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:22.175+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:22.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:22.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:22.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:22.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:22.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:22.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:22.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:22.213+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:22.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:22.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:22.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:22.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:22.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:22.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:22.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:22.507+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:23.489 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:23.256+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16454 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:23.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:23.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:23.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:23.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:23.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:23.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:23.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:23.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:23.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:23.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:23.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:23.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:23.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:23.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:23.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:23.179+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:23.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:23.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:23.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:23.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:23.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:23.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:23.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:23.481+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:24.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:24.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:24.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:24.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:24.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:24.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:24.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:24.214+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:24.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:24.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:24.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:24.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:24.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:24.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:24.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:24.201+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:24.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:24.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:24.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:24.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:24.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:24.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:24.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:24.508+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:25.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:25.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:25.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:25.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:25.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:25.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:25.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:25.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:25.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:25.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:25.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:25.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:25.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:25.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:25.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:25.225+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:25.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:25.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:25.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:25.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:25.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:25.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:25.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:25.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:26.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:26.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:26.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:26.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:26.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:26.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:26.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:26.188+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:26.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:26.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:26.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:26.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:26.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:26.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:26.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:26.253+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:26.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:26.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:26.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:26.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:26.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:26.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:26.552 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:26.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:27.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:27.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:27.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:27.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:27.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:27.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:27.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:27.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:27.402 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:27.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:27.402 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:27.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:27.403 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:27.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:27.403 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:27.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:27.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:27.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:27.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:27.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:27.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:27.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:27.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:27.393+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:28.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:28.258+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16540 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:28.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:28.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:28.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:28.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:28.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:28.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:28.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:28.300+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:28.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:28.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:28.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:28.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:28.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:28.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:28.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:28.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:28.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:28.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:28.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:28.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:28.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:28.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:28.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:28.372+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:29.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:29.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:29.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:29.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:29.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:29.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:29.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:29.279+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:29.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:29.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:29.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:29.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:29.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:29.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:29.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:29.064+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:29.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:29.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:29.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:29.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:29.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:29.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:29.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:29.420+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:30.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:30.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:30.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:30.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:30.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:30.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:30.385 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:30.294+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:30.386 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:30.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:30.386 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:30.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:30.386 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:30.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:30.386 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:30.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:30.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:30.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:30.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:30.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:30.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:30.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:30.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:30.376+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:31.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:31.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:31.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:31.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:31.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:31.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:31.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:31.263+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:31.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:31.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:31.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:31.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:31.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:31.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:31.391 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:31.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:31.391 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:31.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:31.391 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:31.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:31.391 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:31.131+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:31.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:31.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:32.349 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:32.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:32.349 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:32.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:32.349 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:32.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:32.349 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:32.084+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:32.349 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:32.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:32.349 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:32.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:32.350 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:32.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:32.350 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:32.228+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:32.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:32.339+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:32.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:32.339+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:32.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:32.339+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:32.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:32.340+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:33.393 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:33.261+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16629 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:33.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:33.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:33.394 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:33.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:33.394 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:33.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:33.394 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:33.293+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:33.394 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:33.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:33.394 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:33.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:33.394 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:33.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:33.395 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:33.223+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:33.395 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:33.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:33.395 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:33.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:33.395 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:33.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:33.395 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:33.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:34.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:34.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:34.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:34.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:34.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:34.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:34.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:34.338+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:34.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:34.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:34.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:34.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:34.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:34.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:34.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:34.230+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:34.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:34.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:34.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:34.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:34.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:34.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:34.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:34.106+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:35.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:35.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:35.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:35.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:35.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:35.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:35.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:35.317+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:35.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:35.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:35.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:35.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:35.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:35.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:35.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:35.193+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:35.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:35.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:35.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:35.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:35.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:35.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:35.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:35.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:36.316 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:36.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:36.316 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:36.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:36.317 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:36.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:36.317 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:36.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:36.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:36.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:36.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:36.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:36.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:36.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:36.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:36.236+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:36.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:36.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:36.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:36.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:36.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:36.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:36.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:36.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:37.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:37.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:37.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:37.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:37.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:37.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:37.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:37.255+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:37.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:37.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:37.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:37.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:37.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:37.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:37.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:37.018+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:37.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:37.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:37.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:37.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:37.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:37.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:37.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:37.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:38.344 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:38.264+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16715 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:38.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:38.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:38.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:38.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:38.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:38.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:38.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:38.207+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:38.345 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:38.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:38.345 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:38.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:38.345 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:38.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:38.345 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:38.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:38.682 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:38.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:38.682 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:38.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:38.682 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:38.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:38.682 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:38.335+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:39.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:39.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:39.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:39.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:39.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:39.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:39.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:39.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:39.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:39.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:39.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:39.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:39.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:39.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:39.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:39.219+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:39.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:39.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:39.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:39.363+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:40.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:40.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:40.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:40.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:40.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:40.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:40.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:40.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:40.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:40.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:40.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:40.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:40.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:40.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:40.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:40.216+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:40.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:40.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:40.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:40.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:40.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:40.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:40.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:40.375+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:41.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:41.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:41.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:41.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:41.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:41.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:41.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:41.078+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:41.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:41.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:41.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:41.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:41.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:41.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:41.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:41.199+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:41.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:41.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:41.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:41.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:41.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:41.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:41.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:41.374+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:42.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:42.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:42.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:42.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:42.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:42.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:42.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:42.226+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:42.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:42.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:42.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:42.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:42.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:42.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:42.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:42.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:42.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:42.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:42.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:42.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:42.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:42.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:42.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:42.361+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:43.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:43.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:43.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:43.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:43.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:43.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:43.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:43.193+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:43.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:43.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:43.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:43.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:43.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:43.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:43.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:43.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:43.550 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:43.266+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16800 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:43.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:43.325+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:43.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:43.325+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:43.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:43.325+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:43.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:43.326+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:44.299 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:44.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:44.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:44.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:44.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:44.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:44.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:44.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:44.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:43.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:44.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:43.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:44.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:43.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:44.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:43.986+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:44.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:44.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:44.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:44.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:44.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:44.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:44.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:44.320+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:45.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:45.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:45.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:45.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:45.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:45.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:45.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:45.016+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:45.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:45.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:45.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:45.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:45.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:45.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:45.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:45.286+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:45.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:45.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:45.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:45.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:45.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:45.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:45.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:45.211+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:46.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:46.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:46.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:46.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:46.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:46.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:46.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:46.276+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:46.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:46.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:46.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:46.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:46.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:46.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:46.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:46.186+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:46.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:46.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:46.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:46.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:46.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:46.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:46.302 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:46.029+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:47.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:47.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:47.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:47.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:47.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:47.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:47.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:47.296+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:47.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:47.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:47.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:47.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:47.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:47.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:47.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:47.170+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:47.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:47.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:47.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:47.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:47.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:47.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:47.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:47.073+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:48.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:48.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:48.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:48.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:48.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:48.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:48.166 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:48.092+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:48.549 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:48.269+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16887 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:48.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:48.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:48.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:48.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:48.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:48.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:48.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:48.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:48.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:48.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:48.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:48.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:48.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:48.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:48.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:48.157+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:49.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:49.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:49.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:49.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:49.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:49.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:49.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:49.307+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:49.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:49.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:49.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:49.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:49.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:49.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:49.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:49.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:49.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:49.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:49.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:49.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:49.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:49.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:49.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:49.186+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:50.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:50.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:50.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:50.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:50.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:50.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:50.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:50.300+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:50.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:50.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:50.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:50.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:50.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:50.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:50.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:50.158+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:50.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:50.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:50.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:50.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:50.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:50.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:50.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:50.149+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:51.264 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:51.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:51.264 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:51.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:51.264 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:51.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:51.264 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:51.119+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:51.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:51.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:51.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:51.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:51.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:51.120+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:51.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:51.121+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:51.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:51.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:51.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:51.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:51.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:51.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:51.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:51.256+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:52.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:52.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:52.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:52.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:52.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:52.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:52.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:52.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:52.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:52.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:52.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:52.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:52.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:52.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:52.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:52.139+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:52.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:52.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:52.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:52.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:52.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:52.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:52.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:52.238+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:53.405 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:53.272+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 16977 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:53.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:53.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:53.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:53.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:53.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:53.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:53.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:53.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:53.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:53.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:53.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:53.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:53.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:53.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:53.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:53.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:53.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:53.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:53.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:53.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:53.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:53.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:53.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:53.145+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:54.299 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:54.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:54.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:54.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:54.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:54.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:54.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:54.232+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:54.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:54.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:54.300 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:54.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:54.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:54.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:54.301 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:54.135+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:54.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:54.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:54.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:54.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:54.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:54.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:54.301 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:54.105+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:55.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:55.209+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:55.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:55.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:55.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:55.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:55.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:55.210+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:55.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:55.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:55.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:55.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:55.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:55.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:55.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:55.157+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:55.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:55.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:55.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:55.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:55.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:55.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:55.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:55.099+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:56.175 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:56.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:56.176 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:56.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:56.176 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:56.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:56.176 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:56.089+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:56.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:56.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:56.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:56.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:56.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:56.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:56.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:56.167+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:56.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:56.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:56.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:56.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:56.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:56.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:56.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:56.215+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:57.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:57.203+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:57.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:57.203+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:57.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:57.203+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:57.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:57.203+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:57.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:57.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:57.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:57.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:57.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:57.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:57.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:57.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:57.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:57.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:57.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:57.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:57.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:57.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:57.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:57.141+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:58.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:22:58.274+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17064 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:22:58.409 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:58.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:58.409 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:58.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:58.409 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:58.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:58.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:58.167+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:58.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:58.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:58.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:58.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:58.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:58.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:58.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:58.122+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:58.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:58.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:58.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:58.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:58.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:58.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:58.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:58.174+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:59.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:59.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:59.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:59.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:59.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:59.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:22:59.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:22:59.164+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:22:59.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:59.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:59.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:59.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:59.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:59.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:22:59.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:22:59.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:22:59.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:59.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:59.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:59.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:59.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:59.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:22:59.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:22:59.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:00.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:00.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:00.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:00.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:00.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:00.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:00.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:00.121+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:00.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:00.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:00.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:00.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:00.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:00.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:00.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:00.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:00.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:00.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:00.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:00.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:00.245+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:01.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:01.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:01.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:01.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:01.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:01.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:01.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:01.134+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:01.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:01.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:01.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:01.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:01.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:01.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:01.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:01.215+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:01.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:01.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:01.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:01.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:01.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:01.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:01.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:01.218+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:02.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:02.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:02.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:02.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:02.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:02.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:02.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:02.148+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:02.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:02.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:02.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:02.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:02.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:02.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:02.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:02.245+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:02.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:02.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:02.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:02.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:02.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:02.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:02.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:02.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:03.549 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:03.277+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17151 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:03.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:03.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:03.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:03.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:03.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:03.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:03.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:03.174+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:03.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:03.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:03.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:03.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:03.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:03.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:03.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:03.229+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:03.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:03.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:03.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:03.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:03.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:03.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:03.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:03.275+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:04.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:04.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:04.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:04.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:04.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:04.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:04.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:04.170+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:04.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:04.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:04.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:04.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:04.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:04.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:04.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:04.257+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:04.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:04.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:04.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:04.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:04.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:04.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:04.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:04.266+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:05.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:05.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:05.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:05.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:05.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:05.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:05.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:05.149+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:05.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:05.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:05.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:05.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:05.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:05.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:05.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:05.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:05.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:05.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:05.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:05.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:05.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:05.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:05.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:05.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:06.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:06.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:06.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:06.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:06.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:06.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:06.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:06.157+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:06.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:06.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:06.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:06.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:06.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:06.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:06.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:06.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:06.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:06.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:06.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:06.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:06.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:06.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:06.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:06.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:07.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:07.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:07.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:07.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:07.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:07.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:07.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:07.146+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:07.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:07.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:07.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:07.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:07.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:07.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:07.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:07.208+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:07.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:07.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:07.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:07.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:07.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:07.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:07.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:07.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:08.396 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:08.280+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17238 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:08.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:08.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:08.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:08.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:08.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:08.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:08.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:08.107+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:08.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:08.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:08.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:08.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:08.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:08.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:08.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:08.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:08.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:08.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:08.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:08.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:08.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:08.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:08.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:08.388+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:09.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:09.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:09.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:09.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:09.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:09.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:09.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:09.250+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:09.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:09.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:09.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:09.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:09.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:09.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:09.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:09.125+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:09.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:09.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:09.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:09.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:09.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:09.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:09.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:09.419+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:10.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:10.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:10.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:10.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:10.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:10.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:10.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:10.153+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:10.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:10.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:10.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:10.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:10.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:10.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:10.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:10.204+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:10.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:10.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:10.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:10.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:10.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:10.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:10.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:10.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:11.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:11.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:11.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:11.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:11.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:11.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:11.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:11.164+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:11.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:11.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:11.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:11.228+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:11.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:11.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:11.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:11.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:11.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:11.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:11.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:11.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:12.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:12.116+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:12.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:12.116+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:12.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:12.116+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:12.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:12.116+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:12.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:12.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:12.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:12.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:12.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:12.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:12.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:12.200+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:12.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:12.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:12.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:12.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:12.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:12.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:12.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:12.462+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:13.417 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:13.283+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17326 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:13.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:13.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:13.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:13.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:13.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:13.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:13.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:13.133+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:13.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:13.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:13.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:13.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:13.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:13.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:13.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:13.184+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:13.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:13.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:13.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:13.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:13.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:13.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:13.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:13.477+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:14.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:14.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:14.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:14.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:14.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:14.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:14.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:14.130+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:14.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:14.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:14.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:14.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:14.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:14.204+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:14.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:14.204+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:14.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:14.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:14.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:14.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:14.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:14.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:14.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:14.496+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:15.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:15.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:15.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:15.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:15.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:15.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:15.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:15.110+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:15.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:15.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:15.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:15.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:15.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:15.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:15.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:15.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:15.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:15.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:15.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:15.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:15.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:15.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:15.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:15.457+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:16.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:16.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:16.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:16.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:16.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:16.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:16.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:16.105+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:16.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:16.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:16.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:16.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:16.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:16.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:16.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:16.167+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:16.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:16.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:16.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:16.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:17.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:17.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:17.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:17.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:17.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:17.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:17.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:17.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:17.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:17.062+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:17.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:17.062+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:17.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:17.062+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:17.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:17.062+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:17.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:17.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:17.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:17.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:17.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:17.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:17.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:17.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:18.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:18.285+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17413 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:18.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:18.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:18.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:18.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:18.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:18.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:18.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:18.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:18.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:18.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:18.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:18.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:18.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:18.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:18.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:18.082+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:18.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:18.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:18.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:18.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:18.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:18.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:18.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:18.468+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:19.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:19.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:19.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:19.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:19.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:19.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:19.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:19.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:19.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:19.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:19.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:19.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:19.050+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:19.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:19.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:19.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:19.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:19.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:19.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:19.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:19.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:20.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:20.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:20.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:20.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:20.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:20.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:20.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:20.069+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:20.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:20.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:20.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:20.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:20.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:20.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:20.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:20.109+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:20.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:20.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:20.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:20.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:20.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:20.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:20.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:20.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:21.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:21.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:21.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:21.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:21.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:21.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:21.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:21.120+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:21.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:21.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:21.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:21.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:21.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:21.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:21.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:21.119+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:21.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:21.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:21.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:21.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:21.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:21.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:21.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:21.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:22.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:22.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:22.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:22.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:22.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:22.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:22.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:22.169+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:22.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:22.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:22.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:22.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:22.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:22.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:22.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:22.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:22.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:22.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:22.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:22.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:22.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:22.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:22.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:22.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:23.316 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:23.288+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17499 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:23.317 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:23.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:23.317 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:23.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:23.317 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:23.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:23.317 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:23.204+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:23.317 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:23.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:23.317 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:23.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:23.318 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:23.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:23.318 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:23.049+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:23.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:23.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:23.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:23.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:23.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:23.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:23.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:23.477+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:24.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:24.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:24.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:24.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:24.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:24.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:24.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:24.256+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:24.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:24.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:24.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:24.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:24.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:24.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:24.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:24.066+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:24.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:24.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:24.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:24.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:24.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:24.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:24.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:24.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:25.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:25.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:25.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:25.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:25.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:25.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:25.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:25.292+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:25.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:25.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:25.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:25.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:25.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:25.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:25.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:25.105+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:25.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:25.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:25.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:25.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:25.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:26.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:26.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:26.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:26.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:26.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:26.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:26.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:26.283+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:26.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:26.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:26.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:26.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:26.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:26.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:26.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:26.061+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:26.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:26.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:26.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:26.495+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:27.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:27.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:27.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:27.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:27.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:27.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:27.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:27.264+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:27.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:27.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:27.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:27.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:27.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:27.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:27.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:27.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:27.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:27.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:27.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:27.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:27.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:27.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:27.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:27.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:28.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:28.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:28.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:28.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:28.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:28.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:28.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:28.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:28.549 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:28.291+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17587 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:28.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:28.272+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:28.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:28.272+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:28.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:28.272+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:28.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:28.272+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:28.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:28.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:28.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:28.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:28.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:28.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:28.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:28.467+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:29.442 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:29.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:29.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:29.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:29.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:29.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:29.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:29.305+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:29.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:29.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:29.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:29.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:29.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:29.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:29.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:29.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:29.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:29.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:29.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:29.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:29.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:29.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:29.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:29.434+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:30.414 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:30.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:30.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:30.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:30.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:30.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:30.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:30.312+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:30.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:30.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:30.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:30.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:30.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:30.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:30.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:30.097+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:30.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:30.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:30.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:30.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:30.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:30.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:30.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:30.406+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:31.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:31.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:31.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:31.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:31.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:31.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:31.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:31.093+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:31.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:31.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:31.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:31.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:31.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:31.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:31.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:31.358+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:31.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:31.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:31.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:31.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:31.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:31.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:31.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:31.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-02-19T19:23:31.997 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-02-19T19:23:31.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:23:31.998 DEBUG:teuthology.orchestra.run.smithi077:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-02-19T19:23:32.034 INFO:tasks.workunit.client.0.smithi077.stderr:make: *** INTERNAL: readdir: Input/output error. Stop. 2024-02-19T19:23:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-02-19T19:23:32.037 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-02-19T19:23:32.037 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-19T19:23:32.041 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi077.front.sepia.ceph.com 2024-02-19T19:23:32.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-19T19:23:32.041 DEBUG:teuthology.orchestra.run.smithi077:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-19T19:23:32.129 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-02-19T19:23:32.129 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-19T19:23:32.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:32.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:32.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:32.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:32.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:32.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:32.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:32.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:32.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:32.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:32.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:32.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:32.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:32.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:32.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:32.380+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:32.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:32.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:32.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:32.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:32.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:33.429 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:33.294+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17674 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:33.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:33.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:33.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:33.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:33.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:33.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:33.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:33.333+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:33.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:33.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:33.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:33.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:33.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:33.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:33.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:33.093+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:33.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:33.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:33.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:33.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:33.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:33.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:33.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:33.446+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:34.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:34.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:34.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:34.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:34.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:34.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:34.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:34.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:34.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:34.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:34.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:34.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:34.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:34.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:34.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:34.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:34.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:34.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:34.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:34.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:34.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:34.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:34.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:34.442+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:35.299 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:35.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:35.299 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:35.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:35.299 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:35.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:35.300 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:35.265+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:35.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:35.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:35.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:35.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:35.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:35.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:35.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:35.036+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:35.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:35.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:35.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:35.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:35.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:35.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:35.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:35.455+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:36.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:36.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:36.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:36.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:36.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:36.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:36.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:36.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:36.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:36.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:36.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:36.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:36.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:36.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:36.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:36.298+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:36.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:36.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:36.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:36.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:36.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:36.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:36.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:36.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:37.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:36.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:37.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:36.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:37.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:36.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:37.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:36.984+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:37.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:37.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:37.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:37.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:37.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:37.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:37.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:37.302+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:37.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:37.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:37.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:37.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:37.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:37.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:37.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:37.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:37.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:37.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:37.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:37.954+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:38.549 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:38.296+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17762 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:38.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:38.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:38.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:38.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:38.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:38.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:38.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:38.445+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:38.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:38.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:38.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:38.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:38.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:38.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:38.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:38.284+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:39.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:38.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:39.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:38.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:39.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:38.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:39.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:38.931+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:39.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:39.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:39.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:39.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:39.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:39.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:39.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:39.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:39.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:39.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:39.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:39.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:39.298+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:40.172 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:39.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:40.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:39.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:40.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:39.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:40.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:39.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:40.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:40.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:40.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:40.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:40.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:40.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:40.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:40.408+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:40.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:40.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:40.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:40.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:40.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:40.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:40.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:40.315+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:41.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:40.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:41.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:40.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:41.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:40.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:41.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:40.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:41.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:41.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:41.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:41.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:41.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:41.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:41.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:41.386+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:41.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:41.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:41.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:41.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:41.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:41.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:41.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:41.291+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:42.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:41.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:42.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:41.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:42.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:41.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:42.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:41.957+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:42.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:42.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:42.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:42.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:42.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:42.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:42.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:42.382+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:42.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:42.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:42.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:42.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:42.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:42.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:42.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:42.303+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:43.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:42.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:43.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:42.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:43.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:42.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:43.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:42.942+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:43.598 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:43.299+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17849 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:43.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:43.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:43.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:43.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:43.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:43.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:43.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:43.379+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:43.599 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:43.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:43.599 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:43.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:43.599 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:43.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:43.599 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:43.350+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:44.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:43.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:44.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:43.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:44.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:43.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:44.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:43.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:44.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:44.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:44.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:44.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:44.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:44.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:44.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:44.369+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:44.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:44.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:44.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:44.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:44.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:44.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:44.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:44.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:45.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:44.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:45.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:44.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:45.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:44.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:45.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:44.955+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:45.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:45.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:45.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:45.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:45.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:45.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:45.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:45.430+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:45.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:45.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:45.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:45.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:45.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:45.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:45.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:45.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:46.264 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:45.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:46.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:45.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:46.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:45.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:46.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:45.949+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:46.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:46.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:46.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:46.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:46.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:46.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:46.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:46.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:46.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:46.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:46.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:46.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:46.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:46.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:46.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:46.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:47.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:46.970+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:47.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:46.970+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:47.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:46.970+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:47.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:46.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:47.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:47.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:47.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:47.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:47.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:47.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:47.653 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:47.395+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:47.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:47.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:47.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:47.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:47.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:47.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:47.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:47.304+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:48.177 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:48.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:48.177 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:48.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:48.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:48.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:48.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:48.011+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:48.435 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:48.301+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 17937 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:48.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:48.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:48.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:48.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:48.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:48.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:48.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:48.383+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:48.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:48.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:48.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:48.346+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:48.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:48.346+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:48.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:48.346+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:49.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:49.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:49.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:49.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:49.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:49.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:49.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:49.031+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:49.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:49.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:49.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:49.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:49.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:49.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:49.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:49.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:49.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:49.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:49.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:49.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:49.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:49.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:49.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:49.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:50.361 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:50.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:50.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:50.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:50.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:50.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:50.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:50.060+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:50.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:50.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:50.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:50.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:50.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:50.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:50.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:50.354+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:50.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:50.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:50.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:50.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:50.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:50.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:50.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:50.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:51.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:51.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:51.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:51.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:51.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:51.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:51.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:51.024+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:51.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:51.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:51.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:51.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:51.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:51.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:51.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:51.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:51.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:51.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:51.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:51.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:51.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:51.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:51.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:51.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:52.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:51.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:52.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:51.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:52.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:51.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:52.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:51.993+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:52.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:52.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:52.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:52.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:52.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:52.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:52.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:52.371+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:52.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:52.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:52.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:52.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:52.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:52.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:52.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:52.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:53.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:53.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:53.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:53.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:53.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:53.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:53.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:53.020+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:53.599 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:53.304+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18024 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:53.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:53.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:53.600 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:53.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:53.600 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:53.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:53.600 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:53.396+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:53.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:53.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:53.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:53.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:53.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:53.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:53.601 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:53.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:54.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:53.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:54.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:53.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:54.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:53.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:54.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:53.991+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:54.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:54.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:54.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:54.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:54.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:54.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:54.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:54.369+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:54.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:54.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:54.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:54.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:54.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:54.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:54.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:54.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:55.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:54.970+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:55.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:54.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:55.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:54.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:55.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:54.971+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:55.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:55.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:55.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:55.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:55.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:55.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:55.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:55.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:55.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:55.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:55.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:55.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:55.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:55.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:55.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:55.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:56.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:55.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:56.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:55.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:56.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:55.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:56.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:55.952+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:56.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:56.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:56.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:56.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:56.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:56.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:56.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:56.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:56.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:56.384+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:56.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:56.384+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:56.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:56.384+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:56.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:56.384+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:57.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:56.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:57.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:56.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:57.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:56.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:57.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:56.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:57.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:57.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:57.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:57.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:57.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:57.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:57.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:57.320+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:57.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:57.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:57.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:57.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:57.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:57.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:57.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:57.390+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:58.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:57.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:58.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:57.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:58.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:57.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:58.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:57.867+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:58.569 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:23:58.307+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18112 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:23:58.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:58.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:58.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:58.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:58.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:58.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:58.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:58.366+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:58.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:58.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:58.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:58.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:58.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:58.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:58.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:58.308+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:59.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:58.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:59.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:58.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:59.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:58.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:23:59.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:58.879+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:23:59.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:59.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:59.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:59.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:59.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:59.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:23:59.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:23:59.321+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:23:59.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:59.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:59.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:59.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:59.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:59.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:23:59.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:23:59.300+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:00.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:59.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:00.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:59.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:00.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:59.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:00.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:23:59.893+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:00.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:00.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:00.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:00.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:00.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:00.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:00.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:00.281+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:00.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:00.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:00.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:00.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:00.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:00.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:00.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:00.322+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:01.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:00.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:01.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:00.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:01.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:00.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:01.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:00.885+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:01.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:01.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:01.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:01.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:01.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:01.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:01.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:01.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:01.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:01.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:01.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:01.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:01.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:01.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:01.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:01.333+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:02.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:01.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:02.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:01.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:02.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:01.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:02.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:01.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:02.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:02.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:02.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:02.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:02.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:02.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:02.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:02.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:02.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:02.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:02.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:02.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:02.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:02.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:02.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:02.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:03.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:02.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:03.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:02.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:03.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:02.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:03.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:02.936+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:03.600 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:03.310+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18198 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:03.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:03.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:03.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:03.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:03.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:03.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:03.601 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:03.320+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:03.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:03.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:03.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:03.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:03.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:03.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:03.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:03.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:04.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:03.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:04.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:03.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:04.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:03.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:04.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:03.980+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:04.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:04.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:04.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:04.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:04.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:04.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:04.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:04.301+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:04.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:04.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:04.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:04.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:04.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:04.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:04.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:04.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:05.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:04.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:05.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:04.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:05.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:04.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:05.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:04.993+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:05.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:05.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:05.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:05.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:05.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:05.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:05.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:05.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:05.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:05.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:05.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:05.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:05.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:05.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:05.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:05.341+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:06.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:06.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:06.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:06.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:06.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:06.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:06.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:06.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:06.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:06.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:06.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:06.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:06.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:06.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:06.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:06.313+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:06.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:06.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:06.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:06.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:06.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:06.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:06.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:06.378+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:07.357 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:07.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:07.357 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:07.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:07.357 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:07.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:07.357 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:07.060+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:07.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:07.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:07.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:07.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:07.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:07.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:07.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:07.350+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:07.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:07.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:07.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:07.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:07.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:07.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:07.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:07.382+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:08.324 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:08.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:08.324 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:08.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:08.324 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:08.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:08.324 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:08.063+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:08.799 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:08.313+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18286 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:08.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:08.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:08.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:08.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:08.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:08.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:08.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:08.390+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:08.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:08.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:08.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:08.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:08.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:08.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:08.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:08.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:09.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:09.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:09.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:09.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:09.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:09.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:09.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:09.112+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:09.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:09.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:09.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:09.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:09.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:09.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:09.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:09.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:09.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:09.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:09.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:09.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:09.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:09.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:09.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:09.391+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:10.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:10.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:10.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:10.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:10.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:10.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:10.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:10.070+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:10.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:10.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:10.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:10.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:10.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:10.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:10.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:10.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:10.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:10.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:10.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:10.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:10.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:10.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:10.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:10.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:11.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:11.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:11.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:11.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:11.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:11.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:11.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:11.073+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:11.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:11.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:11.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:11.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:11.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:11.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:11.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:11.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:11.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:11.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:11.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:11.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:11.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:11.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:11.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:11.380+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:12.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:12.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:12.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:12.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:12.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:12.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:12.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:12.033+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:12.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:12.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:12.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:12.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:12.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:12.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:12.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:12.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:12.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:12.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:12.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:12.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:12.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:12.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:12.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:12.383+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:13.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:13.316+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18373 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:13.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:13.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:13.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:13.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:13.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:13.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:13.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:13.068+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:13.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:13.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:13.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:13.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:13.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:13.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:13.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:13.391+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:13.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:13.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:13.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:13.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:13.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:13.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:13.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:13.407+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:14.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:14.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:14.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:14.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:14.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:14.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:14.299 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:14.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:14.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:14.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:14.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:14.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:14.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:14.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:14.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:14.422+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:14.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:14.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:14.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:14.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:14.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:14.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:14.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:14.445+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:15.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:15.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:15.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:15.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:15.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:15.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:15.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:15.051+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:15.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:15.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:15.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:15.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:15.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:15.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:15.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:15.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:15.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:15.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:15.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:15.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:15.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:15.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:15.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:15.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:16.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:16.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:16.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:16.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:16.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:16.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:16.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:16.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:16.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:16.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:16.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:16.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:16.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:16.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:16.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:16.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:17.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:17.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:17.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:17.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:17.379 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:17.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:17.379 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:17.102+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:17.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:17.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:17.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:17.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:17.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:17.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:17.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:17.385+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:17.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:17.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:17.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:17.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:17.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:17.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:17.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:17.371+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:18.406 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:18.319+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18461 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:18.407 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:18.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:18.407 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:18.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:18.407 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:18.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:18.407 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:18.144+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:18.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:18.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:18.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:18.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:18.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:18.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:18.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:18.390+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:18.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:18.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:18.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:18.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:18.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:18.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:18.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:18.399+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:19.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:19.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:19.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:19.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:19.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:19.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:19.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:19.430+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:19.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:19.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:19.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:19.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:19.183+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:19.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:19.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:19.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:19.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:19.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:19.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:19.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:19.359+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:20.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:20.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:20.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:20.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:20.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:20.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:20.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:20.147+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:20.473 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:20.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:20.473 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:20.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:20.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:20.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:20.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:20.386+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:20.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:20.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:20.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:20.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:20.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:20.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:20.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:20.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:21.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:21.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:21.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:21.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:21.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:21.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:21.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:21.171+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:21.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:21.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:21.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:21.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:21.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:21.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:21.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:21.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:21.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:21.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:21.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:21.441+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:22.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:22.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:22.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:22.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:22.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:22.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:22.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:22.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:22.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:22.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:22.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:22.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:22.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:22.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:22.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:22.443+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:22.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:22.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:22.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:22.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:22.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:22.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:22.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:22.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:23.420 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:23.322+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18547 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:23.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:23.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:23.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:23.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:23.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:23.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:23.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:23.136+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:23.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:23.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:23.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:23.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:23.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:23.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:23.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:23.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:23.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:23.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:23.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:23.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:23.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:23.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:23.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:24.468 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:24.468 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:24.468 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:24.469 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:24.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:24.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:24.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:24.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:24.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:24.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:24.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:24.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:24.499+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:24.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:24.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:24.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:24.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:25.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:25.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:25.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:25.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:25.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:25.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:25.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:25.202+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:25.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:25.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:25.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:25.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:25.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:25.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:25.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:25.487+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:25.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:25.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:25.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:25.508+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:26.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:26.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:26.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:26.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:26.481 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:26.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:26.481 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:26.163+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:26.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:26.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:26.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:26.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:26.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:26.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:26.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:26.516+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:26.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:26.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:26.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:26.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:27.441 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:27.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:27.441 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:27.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:27.441 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:27.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:27.441 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:27.183+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:27.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:27.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:27.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:27.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:27.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:27.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:27.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:27.434+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:27.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:27.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:27.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:27.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:27.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:27.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:27.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:27.565+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:28.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:28.325+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18635 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:28.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:28.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:28.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:28.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:28.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:28.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:28.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:28.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:28.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:28.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:28.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:28.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:28.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:28.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:28.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:28.228+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:29.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:28.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:29.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:28.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:29.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:28.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:29.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:28.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:29.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:29.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:29.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:29.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:29.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:29.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:29.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:29.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:29.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:29.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:29.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:29.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:29.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:29.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:29.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:29.264+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:30.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:29.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:30.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:29.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:30.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:29.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:30.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:29.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:30.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:30.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:30.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:30.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:30.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:30.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:30.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:30.259+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:30.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:30.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:30.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:30.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:30.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:30.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:30.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:30.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:30.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:30.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:30.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:30.501+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:31.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:31.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:31.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:31.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:31.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:31.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:31.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:31.462+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:31.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:31.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:31.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:31.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:31.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:31.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:31.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:31.442+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:31.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:31.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:31.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:31.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:31.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:31.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:31.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:31.250+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:32.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:32.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:32.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:32.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:32.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:32.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:32.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:32.461+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:32.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:32.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:32.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:32.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:32.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:32.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:32.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:32.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:32.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:32.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:32.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:32.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:32.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:32.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:32.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:32.270+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:33.600 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:33.328+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18723 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:33.601 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:33.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:33.601 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:33.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:33.601 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:33.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:33.601 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:33.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:33.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:33.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:33.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:33.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:33.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:33.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:33.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:33.299+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:33.602 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:33.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:33.602 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:33.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:33.602 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:33.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:33.602 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:33.466+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:34.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:34.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:34.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:34.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:34.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:34.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:34.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:34.498+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:34.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:34.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:34.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:34.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:34.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:34.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:34.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:34.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:34.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:34.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:34.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:34.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:34.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:34.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:34.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:34.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:35.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:35.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:35.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:35.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:35.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:35.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:35.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:35.499+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:35.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:35.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:35.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:35.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:35.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:35.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:35.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:35.266+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:35.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:35.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:35.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:35.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:35.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:35.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:35.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:35.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:36.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:36.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:36.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:36.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:36.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:36.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:36.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:36.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:36.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:36.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:36.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:36.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:36.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:36.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:36.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:36.508+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:36.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:36.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:36.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:36.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:36.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:36.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:36.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:36.483+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:37.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:37.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:37.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:37.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:37.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:37.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:37.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:37.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:37.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:37.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:37.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:37.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:37.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:37.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:37.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:37.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:37.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:37.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:37.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:37.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:37.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:37.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:37.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:37.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:38.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:38.331+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18812 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:38.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:38.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:38.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:38.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:38.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:38.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:38.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:38.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:38.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:38.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:38.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:38.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:38.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:38.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:38.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:38.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:38.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:38.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:38.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:38.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:38.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:38.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:38.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:38.485+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:39.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:39.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:39.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:39.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:39.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:39.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:39.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:39.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:39.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:39.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:39.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:39.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:39.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:39.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:39.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:39.235+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:39.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:39.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:39.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:39.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:39.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:39.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:39.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:39.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:40.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:40.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:40.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:40.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:40.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:40.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:40.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:40.231+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:40.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:40.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:40.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:40.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:40.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:40.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:40.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:40.425+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:40.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:40.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:40.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:40.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:40.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:40.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:40.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:40.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:41.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:41.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:41.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:41.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:41.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:41.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:41.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:41.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:41.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:41.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:41.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:41.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:41.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:41.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:41.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:41.243+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:42.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:41.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:42.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:41.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:42.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:41.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:42.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:41.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:42.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:42.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:42.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:42.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:42.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:42.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:42.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:42.415+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:42.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:42.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:42.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:42.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:42.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:42.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:42.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:42.202+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:43.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:42.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:43.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:42.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:43.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:42.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:43.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:42.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:43.458 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:43.335+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 18931 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:43.458 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:43.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:43.458 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:43.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:43.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:43.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:43.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:43.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:43.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:43.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:43.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:43.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:43.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:43.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:43.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:43.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:43.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:43.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:43.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:43.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:43.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:43.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:43.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:43.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:44.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:44.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:44.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:44.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:44.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:44.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:44.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:44.189+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:44.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:44.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:44.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:44.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:44.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:44.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:44.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:44.477+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:44.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:44.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:44.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:44.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:44.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:44.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:44.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:44.498+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:45.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:45.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:45.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:45.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:45.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:45.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:45.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:45.146+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:45.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:45.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:45.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:45.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:45.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:45.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:45.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:45.515+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:45.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:45.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:45.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:45.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:45.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:45.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:45.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:45.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:46.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:46.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:46.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:46.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:46.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:46.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:46.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:46.158+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:46.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:46.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:46.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:46.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:46.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:46.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:46.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:46.512+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:46.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:46.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:46.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:46.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:46.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:46.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:46.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:46.545+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:47.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:47.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:47.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:47.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:47.192+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:47.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:47.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:47.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:47.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:47.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:47.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:47.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:47.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:47.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:47.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:47.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:47.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:47.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:47.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:47.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:47.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:48.548 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:48.337+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19019 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:48.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:48.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:48.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:48.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:48.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:48.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:48.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:48.441+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:48.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:48.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:48.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:48.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:48.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:48.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:48.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:48.236+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:49.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:48.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:49.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:48.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:49.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:48.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:49.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:48.558+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:49.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:49.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:49.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:49.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:49.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:49.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:49.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:49.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:49.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:49.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:49.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:49.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:49.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:49.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:49.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:49.278+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:49.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:49.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:49.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:49.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:49.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:49.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:49.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:49.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:50.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:50.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:50.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:50.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:50.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:50.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:16.786604+0000 (oldest deadline 2024-02-19T19:14:36.786604+0000) 2024-02-19T19:24:50.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:50.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:50.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:50.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:50.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:50.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:50.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:50.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:50.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:50.283+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:51.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:50.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:51.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:50.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:51.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:50.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:14.173190+0000 (oldest deadline 2024-02-19T19:14:34.173190+0000) 2024-02-19T19:24:51.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:50.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:51.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:51.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:51.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:51.396+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:51.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:51.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:51.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:51.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:51.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:51.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:51.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:51.325+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:52.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:52.521+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:52.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:52.424+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:52.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:52.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:52.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:52.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:52.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:52.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:52.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:52.337+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:53.602 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:53.406+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:53.603 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:53.340+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19106 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:53.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:53.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:53.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:53.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:53.604 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:53.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:53.604 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:53.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:53.604 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:53.371+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:54.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:54.383+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:54.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:54.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:54.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:54.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:54.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:54.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:54.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:54.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:54.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:54.377+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:55.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:55.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:55.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:55.531+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:55.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:55.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:55.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:55.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:55.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:55.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:55.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:55.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:56.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:56.398+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:56.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:56.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:56.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:56.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:56.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:56.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:56.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:56.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:56.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:56.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:57.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:57.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:57.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:57.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:57.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:57.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:57.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:57.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:57.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:57.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:58.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:57.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:58.616 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:24:58.343+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19193 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:24:58.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:58.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:58.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:58.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:58.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:58.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:58.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:58.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:14:24.642969+0000 (oldest deadline 2024-02-19T19:14:44.642969+0000) 2024-02-19T19:24:58.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:58.308+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:24:59.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:58.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:59.574 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:24:59.361+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:24:59.575 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:24:59.305+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:00.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:24:59.567+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:00.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:00.396+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:00.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:00.309+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:01.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:00.560+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:01.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:01.391+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:01.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:01.321+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:02.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:01.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:02.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:02.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:02.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:02.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:02.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:02.354+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:03.603 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:03.346+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19278 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:03.604 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:03.462+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:03.604 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:03.315+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:03.605 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:03.503+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:04.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:04.336+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:04.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:04.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:04.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:04.547+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:05.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:05.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:05.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:05.497+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:05.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:05.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:06.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:06.515+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:06.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:06.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:06.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:06.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:07.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:07.408+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:07.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:07.532+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:07.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:07.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:08.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:08.549+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:08.610 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:08.349+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19359 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:08.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:08.437+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:09.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:08.602+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:09.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:09.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:09.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:09.398+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:09.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:09.567+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:10.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:10.414+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:10.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:10.519+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:10.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:10.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:11.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:11.459+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:11.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:11.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:11.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:11.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:12.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:12.456+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:12.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:12.424+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:13.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:12.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:13.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:13.400+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:13.798 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:13.352+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19446 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:13.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:13.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:13.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:13.578+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:14.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:14.421+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:14.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:14.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:14.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:14.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:14.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:14.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:14.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:14.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:14.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:14.623+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:15.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:15.410+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:15.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:15.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:15.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:15.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:15.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:15.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:15.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:15.420+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:16.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:15.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:16.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:15.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:16.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:15.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:16.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:15.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:16.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:16.377+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:16.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:16.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:16.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:16.424+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:16.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:16.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:16.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:16.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:16.561+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:17.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:17.403+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:17.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:17.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:17.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:17.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:17.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:17.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:17.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:17.399+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:17.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:17.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:17.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:17.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:17.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:17.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:17.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:17.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:18.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:18.355+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19528 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:18.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:18.364+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:18.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:18.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:18.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:18.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:18.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:18.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:18.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:18.426+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:18.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:18.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:18.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:18.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:18.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:18.548+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:18.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:18.548+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:19.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:19.404+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:19.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:19.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:19.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:19.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:19.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:19.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:19.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:19.397+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:19.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:19.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:19.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:19.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:19.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:19.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:19.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:19.566+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:20.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:20.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:20.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:20.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:20.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:20.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:20.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:20.400+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:20.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:20.448+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:20.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:20.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:20.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:20.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:20.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:20.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:20.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:20.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:21.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:21.425+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:21.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:21.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:21.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:21.424+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:21.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:21.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:21.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:21.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:21.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:21.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:21.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:21.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:22.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:22.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:22.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:22.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:22.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:22.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:22.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:22.429+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:22.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:22.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:22.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:22.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:22.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:22.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:22.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:22.510+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:22.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:22.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:22.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:22.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:22.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:22.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:22.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:22.474+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:23.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:23.358+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19614 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:23.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:23.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:23.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:23.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:23.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:23.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:23.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:23.455+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:23.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:23.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:23.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:23.521+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:23.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:23.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:23.800 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:23.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:23.800 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:23.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:23.800 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:23.430+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:24.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:24.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:24.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:24.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:24.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:24.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:24.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:24.475+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:24.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:24.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:24.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:24.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:24.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:24.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:24.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:24.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:24.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:24.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:24.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:24.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:25.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:25.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:25.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:25.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:25.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:25.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:25.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:25.426+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:25.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:25.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:25.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:25.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:25.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:25.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:25.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:25.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:25.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:25.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:25.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:25.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:26.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:26.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:26.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:26.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:26.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:26.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:26.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:26.476+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:26.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:26.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:26.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:26.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:26.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:26.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:26.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:26.335+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:26.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:26.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:26.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:26.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:26.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:26.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:26.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:26.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:27.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:27.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:27.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:27.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:27.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:27.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:27.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:27.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:27.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:27.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:27.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:27.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:27.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:27.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:27.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:27.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:27.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:27.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:27.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:27.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:27.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:27.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:27.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:27.339+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:28.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:28.361+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19701 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:28.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:28.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:28.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:28.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:28.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:28.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:28.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:28.358+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:28.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:28.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:28.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:28.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:28.458+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:28.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:28.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:28.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:28.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:28.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:28.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:28.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:28.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:29.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:29.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:29.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:29.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:29.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:29.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:29.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:29.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:29.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:29.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:29.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:29.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:29.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:29.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:29.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:29.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:29.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:29.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:29.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:29.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:29.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:29.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:29.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:29.443+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:30.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:30.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:30.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:30.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:30.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:30.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:30.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:30.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:30.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:30.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:30.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:30.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:30.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:30.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:30.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:30.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:30.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:30.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:30.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:30.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:30.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:30.560+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:31.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:31.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:31.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:31.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:31.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:31.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:31.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:31.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:31.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:31.339+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:31.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:31.339+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:31.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:31.339+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:31.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:31.340+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:31.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:31.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:31.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:31.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:31.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:31.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:31.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:31.452+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:32.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:32.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:32.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:32.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:32.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:32.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:32.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:32.586+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:32.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:32.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:32.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:32.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:32.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:32.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:32.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:32.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:32.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:32.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:32.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:32.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:32.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:32.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:32.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:32.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:33.604 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:33.364+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19788 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:33.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:33.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:33.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:33.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:33.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:33.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:33.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:33.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:33.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:33.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:33.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:33.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:33.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:33.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:33.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:33.385+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:34.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:33.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:34.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:33.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:34.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:33.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:34.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:33.626+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:34.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:34.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:34.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:34.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:34.390+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:34.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:34.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:34.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:34.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:34.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:34.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:34.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:34.353+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:35.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:34.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:35.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:34.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:35.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:34.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:35.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:34.644+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:35.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:35.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:35.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:35.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:35.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:35.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:35.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:35.399+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:35.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:35.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:35.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:35.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:35.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:35.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:35.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:35.401+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:36.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:35.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:36.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:35.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:36.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:35.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:36.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:35.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:36.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:36.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:36.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:36.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:36.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:36.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:36.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:36.406+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:36.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:36.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:36.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:36.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:36.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:36.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:36.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:36.383+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:36.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:36.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:37.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:36.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:37.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:36.663+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:37.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:37.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:37.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:37.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:37.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:37.424+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:37.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:37.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:37.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:37.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:37.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:37.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:37.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:37.629+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:37.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:37.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:37.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:37.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:37.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:37.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:37.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:37.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:38.700 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:38.368+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19877 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:38.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:38.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:38.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:38.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:38.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:38.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:38.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:38.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:38.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:38.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:38.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:38.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:38.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:38.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:38.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:38.590+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:38.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:38.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:38.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:38.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:38.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:38.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:38.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:38.449+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:39.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:39.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:39.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:39.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:39.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:39.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:39.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:39.542+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:39.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:39.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:39.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:39.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:39.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:39.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:39.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:39.450+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:39.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:39.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:39.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:39.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:39.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:39.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:39.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:39.369+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:40.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:40.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:40.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:40.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:40.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:40.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:40.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:40.391+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:40.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:40.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:40.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:40.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:40.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:40.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:40.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:40.500+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:40.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:40.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:40.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:40.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:40.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:40.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:40.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:40.558+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:41.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:41.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:41.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:41.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:41.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:41.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:41.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:41.527+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:41.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:41.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:41.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:41.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:41.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:41.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:41.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:41.405+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:41.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:41.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:41.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:41.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:41.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:41.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:41.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:41.550+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:42.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:42.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:42.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:42.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:42.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:42.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:42.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:42.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:42.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:42.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:42.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:42.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:42.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:42.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:42.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:42.398+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:42.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:42.551+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:42.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:42.551+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:42.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:42.551+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:42.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:42.552+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:43.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:43.371+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 19964 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:43.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:43.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:43.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:43.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:43.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:43.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:43.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:43.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:43.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:43.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:43.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:43.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:43.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:43.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:43.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:43.535+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:43.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:43.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:43.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:43.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:43.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:43.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:43.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:43.577+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:44.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:44.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:44.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:44.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:44.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:44.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:44.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:44.378+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:44.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:44.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:44.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:44.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:44.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:44.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:44.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:44.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:44.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:44.551+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:44.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:44.551+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:44.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:44.551+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:44.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:44.551+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:45.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:45.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:45.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:45.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:45.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:45.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:45.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:45.530+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:45.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:45.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:45.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:45.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:45.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:45.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:45.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:45.482+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:45.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:45.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:45.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:45.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:45.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:45.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:45.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:45.394+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:46.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:46.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:46.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:46.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:46.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:46.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:46.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:46.566+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:46.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:46.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:46.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:46.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:46.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:46.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:46.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:46.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:46.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:46.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:46.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:46.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:46.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:46.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:46.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:46.360+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:47.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:47.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:47.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:47.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:47.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:47.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:47.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:47.519+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:47.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:47.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:47.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:47.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:47.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:47.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:47.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:47.393+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:47.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:47.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:47.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:47.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:47.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:47.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:47.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:47.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:48.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:48.374+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20053 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:48.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:48.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:48.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:48.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:48.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:48.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:48.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:48.500+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:48.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:48.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:48.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:48.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:48.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:48.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:48.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:48.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:48.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:48.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:48.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:48.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:48.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:48.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:48.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:48.362+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:49.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:49.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:49.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:49.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:49.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:49.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:49.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:49.487+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:49.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:49.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:49.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:49.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:49.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:49.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:49.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:49.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:49.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:49.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:49.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:49.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:49.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:49.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:49.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:49.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:50.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:50.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:50.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:50.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:50.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:50.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:50.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:50.382+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:50.452 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:50.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:50.452 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:50.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:50.452 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:50.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:50.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:50.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:50.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:50.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:50.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:50.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:50.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:50.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:50.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:50.445+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:51.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:51.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:51.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:51.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:51.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:51.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:51.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:51.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:51.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:51.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:51.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:51.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:51.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:51.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:51.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:51.337+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:51.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:51.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:51.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:51.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:51.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:51.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:51.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:51.417+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:52.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:52.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:52.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:52.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:52.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:52.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:52.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:52.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:52.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:52.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:52.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:52.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:52.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:52.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:52.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:52.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:52.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:52.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:52.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:52.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:52.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:52.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:52.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:52.390+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:53.800 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:53.378+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20142 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:53.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:53.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:53.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:53.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:53.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:53.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:53.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:53.362+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:53.801 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:53.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:53.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:53.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:53.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:53.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:53.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:53.423+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:53.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:53.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:53.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:53.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:53.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:53.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:53.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:53.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:54.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:54.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:54.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:54.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:54.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:54.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:54.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:54.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:54.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:54.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:54.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:54.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:54.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:54.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:54.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:54.437+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:54.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:54.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:54.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:54.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:54.382+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:55.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:55.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:55.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:55.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:55.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:55.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:55.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:55.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:55.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:55.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:55.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:55.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:55.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:55.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:55.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:55.361+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:55.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:55.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:55.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:55.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:55.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:55.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:55.549 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:55.422+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:56.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:56.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:56.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:56.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:56.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:56.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:56.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:56.341+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:56.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:56.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:56.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:56.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:56.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:56.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:56.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:56.383+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:56.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:56.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:56.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:56.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:56.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:56.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:56.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:56.386+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:57.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:57.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:57.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:57.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:57.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:57.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:57.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:57.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:57.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:57.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:57.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:57.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:57.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:57.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:57.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:57.420+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:57.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:57.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:57.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:57.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:57.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:57.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:57.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:57.359+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:58.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:25:58.381+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20232 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:25:58.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:58.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:58.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:58.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:58.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:58.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:58.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:58.357+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:58.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:58.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:58.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:58.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:58.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:58.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:58.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:58.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:58.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:58.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:58.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:58.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:58.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:58.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:58.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:58.439+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:59.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:59.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:59.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:25:59.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:25:59.411+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:25:59.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:59.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:59.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:59.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:59.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:59.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:25:59.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:25:59.365+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:25:59.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:59.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:59.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:59.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:59.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:59.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:25:59.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:25:59.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:00.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:00.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:00.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:00.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:00.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:00.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:00.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:00.403+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:00.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:00.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:00.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:00.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:00.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:00.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:00.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:00.375+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:00.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:00.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:00.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:00.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:00.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:00.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:00.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:00.324+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:01.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:01.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:01.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:01.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:01.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:01.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:01.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:01.450+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:01.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:01.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:01.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:01.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:01.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:01.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:01.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:01.388+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:01.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:01.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:01.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:01.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:01.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:01.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:01.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:01.335+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:02.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:02.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:02.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:02.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:02.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:02.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:02.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:02.451+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:02.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:02.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:02.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:02.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:02.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:02.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:02.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:02.287+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:02.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:02.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:02.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:02.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:02.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:02.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:02.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:02.406+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:03.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:03.384+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20321 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:03.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:03.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:03.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:03.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:03.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:03.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:03.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:03.405+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:03.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:03.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:03.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:03.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:03.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:03.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:03.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:03.364+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:03.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:03.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:03.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:03.261+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:04.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:04.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:04.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:04.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:04.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:04.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:04.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:04.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:04.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:04.388+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:04.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:04.388+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:04.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:04.388+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:04.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:04.389+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:04.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:04.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:04.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:04.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:04.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:04.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:04.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:04.271+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:05.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:05.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:05.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:05.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:05.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:05.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:05.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:05.390+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:05.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:05.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:05.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:05.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:05.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:05.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:05.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:05.245+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:05.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:05.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:05.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:05.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:05.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:05.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:05.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:05.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:06.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:06.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:06.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:06.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:06.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:06.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:06.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:06.418+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:06.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:06.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:06.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:06.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:06.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:06.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:06.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:06.217+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:06.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:06.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:06.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:06.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:06.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:06.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:06.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:06.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:07.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:07.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:07.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:07.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:07.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:07.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:07.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:07.383+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:07.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:07.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:07.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:07.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:07.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:07.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:07.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:07.403+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:07.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:07.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:07.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:07.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:07.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:07.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:07.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:07.228+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:08.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:08.388+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20411 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:08.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:08.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:08.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:08.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:08.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:08.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:08.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:08.440+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:08.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:08.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:08.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:08.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:08.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:08.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:08.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:08.402+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:08.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:08.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:08.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:08.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:08.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:08.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:08.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:08.219+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:09.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:09.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:09.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:09.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:09.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:09.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:09.548 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:09.452+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:09.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:09.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:09.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:09.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:09.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:09.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:09.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:09.256+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:09.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:09.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:09.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:09.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:09.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:09.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:09.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:09.398+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:10.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:10.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:10.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:10.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:10.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:10.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:10.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:10.303+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:10.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:10.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:10.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:10.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:10.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:10.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:10.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:10.433+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:10.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:10.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:10.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:10.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:10.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:10.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:10.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:10.503+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:11.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:11.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:11.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:11.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:11.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:11.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:11.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:11.396+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:11.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:11.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:11.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:11.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:11.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:11.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:11.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:11.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:11.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:11.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:11.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:11.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:11.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:11.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:11.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:11.473+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:12.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:12.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:12.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:12.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:12.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:12.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:12.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:12.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:12.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:12.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:12.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:12.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:12.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:12.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:12.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:12.378+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:12.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:12.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:12.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:12.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:12.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:12.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:12.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:12.459+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:13.719 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:13.391+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20501 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:13.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:13.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:13.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:13.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:13.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:13.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:13.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:13.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:13.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:13.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:13.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:13.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:13.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:13.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:13.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:13.479+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:13.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:13.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:13.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:13.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:13.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:13.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:13.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:13.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:14.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:14.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:14.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:14.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:14.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:14.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:14.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:14.386+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:14.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:14.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:14.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:14.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:14.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:14.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:14.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:14.441+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:14.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:14.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:14.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:14.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:14.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:14.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:14.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:14.470+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:15.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:15.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:15.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:15.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:15.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:15.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:15.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:15.399+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:15.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:15.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:15.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:15.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:15.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:15.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:15.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:15.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:15.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:15.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:15.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:15.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:15.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:15.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:15.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:15.493+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:16.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:16.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:16.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:16.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:16.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:16.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:16.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:16.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:16.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:16.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:16.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:16.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:16.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:16.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:16.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:16.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:16.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:16.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:16.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:16.500+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:17.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:17.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:17.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:17.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:17.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:17.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:17.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:17.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:17.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:17.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:17.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:17.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:17.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:17.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:17.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:17.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:17.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:17.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:17.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:17.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:17.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:17.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:17.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:17.509+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:18.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:18.394+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20591 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:18.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:18.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:18.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:18.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:18.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:18.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:18.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:18.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:18.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:18.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:18.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:18.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:18.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:18.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:18.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:18.516+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:18.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:18.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:18.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:18.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:18.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:18.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:18.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:18.557+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:19.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:19.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:19.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:19.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:19.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:19.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:19.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:19.391+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:19.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:19.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:19.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:19.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:19.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:19.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:19.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:19.555+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:19.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:19.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:19.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:19.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:19.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:19.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:19.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:19.535+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:20.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:20.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:20.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:20.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:20.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:20.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:20.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:20.371+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:20.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:20.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:20.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:20.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:20.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:20.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:20.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:20.541+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:20.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:20.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:20.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:20.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:20.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:20.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:20.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:20.520+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:21.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:21.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:21.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:21.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:21.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:21.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:21.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:21.397+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:21.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:21.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:21.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:21.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:21.506+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:21.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:21.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:21.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:21.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:21.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:21.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:21.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:21.490+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:22.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:22.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:22.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:22.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:22.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:22.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:22.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:22.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:22.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:22.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:22.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:22.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:22.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:22.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:22.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:22.466+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:22.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:22.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:22.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:22.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:22.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:22.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:22.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:22.443+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:23.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:23.398+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20680 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:23.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:23.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:23.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:23.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:23.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:23.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:23.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:23.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:23.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:23.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:23.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:23.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:23.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:23.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:23.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:23.460+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:23.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:23.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:23.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:23.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:23.393+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:24.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:24.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:24.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:24.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:24.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:24.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:24.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:24.439+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:24.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:24.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:24.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:24.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:24.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:24.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:24.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:24.509+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:24.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:24.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:24.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:24.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:24.350+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:25.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:25.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:25.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:25.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:25.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:25.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:25.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:25.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:25.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:25.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:25.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:25.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:25.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:25.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:25.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:25.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:25.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:25.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:25.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:25.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:25.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:25.388+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:26.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:26.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:26.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:26.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:26.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:26.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:26.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:26.388+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:26.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:26.523+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:26.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:26.523+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:26.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:26.523+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:26.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:26.524+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:26.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:26.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:26.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:26.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:26.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:26.386+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:27.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:27.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:27.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:27.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:27.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:27.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:27.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:27.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:27.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:27.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:27.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:27.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:27.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:27.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:27.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:27.359+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:27.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:27.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:27.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:27.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:27.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:27.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:27.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:27.478+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:28.800 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:28.401+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20768 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:28.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:28.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:28.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:28.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:28.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:28.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:28.801 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:28.445+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:28.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:28.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:28.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:28.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:28.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:28.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:28.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:28.378+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:28.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:28.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:28.802 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:28.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:28.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:28.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:28.803 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:28.520+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:29.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:29.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:29.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:29.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:29.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:29.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:29.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:29.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:29.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:29.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:29.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:29.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:29.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:29.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:29.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:29.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:29.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:29.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:29.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:29.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:29.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:29.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:29.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:29.565+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:30.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:30.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:30.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:30.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:30.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:30.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:30.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:30.430+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:30.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:30.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:30.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:30.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:30.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:30.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:30.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:30.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:30.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:30.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:30.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:30.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:30.569+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:31.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:31.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:31.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:31.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:31.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:31.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:31.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:31.440+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:31.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:31.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:31.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:31.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:31.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:31.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:31.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:31.615+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:31.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:31.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:31.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:31.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:31.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:31.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:31.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:31.453+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:32.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:32.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:32.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:32.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:32.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:32.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:32.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:32.452+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:32.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:32.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:32.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:32.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:32.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:32.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:32.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:32.599+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:32.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:32.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:32.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:32.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:32.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:32.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:32.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:32.478+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:33.797 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:33.404+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20859 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:33.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:33.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:33.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:33.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:33.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:33.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:33.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:33.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:33.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:33.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:33.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:33.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:33.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:33.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:33.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:33.526+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:33.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:33.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:33.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:33.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:33.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:33.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:33.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:33.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:34.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:34.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:34.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:34.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:34.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:34.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:34.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:34.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:34.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:34.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:34.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:34.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:34.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:34.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:34.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:34.513+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:34.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:34.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:34.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:34.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:34.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:34.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:34.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:34.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:35.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:35.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:35.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:35.452+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:35.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:35.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:35.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:35.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:35.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:35.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:35.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:35.537+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:35.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:35.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:35.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:35.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:35.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:35.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:35.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:35.576+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:36.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:36.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:36.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:36.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:36.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:36.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:36.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:36.508+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:36.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:36.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:36.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:36.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:36.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:36.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:36.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:36.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:36.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:36.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:36.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:36.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:36.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:36.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:36.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:36.610+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:37.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:37.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:37.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:37.490+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:37.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:37.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:37.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:37.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:37.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:37.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:37.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:37.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:37.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:37.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:37.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:37.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:37.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:37.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:37.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:37.393+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:38.672 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:38.408+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 20947 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:38.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:38.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:38.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:38.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:38.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:38.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:38.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:38.346+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:38.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:38.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:38.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:38.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:38.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:38.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:38.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:38.476+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:39.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:39.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:39.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:39.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:38.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:39.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:39.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:39.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:39.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:39.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:39.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:39.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:39.376+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:39.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:39.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:39.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:39.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:39.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:39.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:39.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:39.460+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:40.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:39.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:40.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:39.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:40.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:39.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:40.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:39.663+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:40.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:40.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:40.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:40.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:40.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:40.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:40.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:40.330+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:40.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:40.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:40.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:40.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:40.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:40.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:40.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:40.689+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:40.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:40.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:40.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:40.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:40.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:40.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:40.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:40.486+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:41.672 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:41.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:41.672 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:41.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:41.672 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:41.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:41.672 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:41.455+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:41.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:41.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:41.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:41.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:41.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:41.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:41.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:41.301+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:42.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:41.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:42.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:41.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:42.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:41.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:42.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:41.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:42.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:42.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:42.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:42.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:42.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:42.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:42.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:42.476+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:42.547 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:42.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:42.547 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:42.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:42.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:42.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:42.548 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:42.285+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:43.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:42.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:42.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:42.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:42.679+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:43.608 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:43.411+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21039 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:43.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:43.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:43.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:43.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:43.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:43.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:43.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:43.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:43.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:43.473+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:43.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:43.473+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:43.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:43.473+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:43.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:43.473+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:44.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:43.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:44.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:43.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:44.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:43.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:44.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:43.660+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:44.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:44.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:44.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:44.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:44.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:44.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:44.705 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:44.497+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:44.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:44.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:44.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:44.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:44.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:44.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:44.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:44.346+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:45.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:44.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:45.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:44.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:45.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:44.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:45.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:44.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:45.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:45.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:45.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:45.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:45.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:45.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:45.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:45.514+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:45.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:45.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:45.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:45.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:45.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:45.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:45.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:45.338+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:46.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:45.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:46.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:45.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:46.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:45.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:46.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:45.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:46.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:46.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:46.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:46.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:46.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:46.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:46.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:46.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:46.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:46.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:46.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:46.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:46.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:46.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:46.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:46.553+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:47.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:46.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:47.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:46.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:47.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:46.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:47.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:46.728+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:47.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:47.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:47.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:47.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:47.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:47.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:47.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:47.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:47.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:47.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:47.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:47.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:47.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:47.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:47.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:47.564+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:48.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:47.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:48.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:47.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:48.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:47.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:48.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:47.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:48.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:48.414+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21128 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:48.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:48.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:48.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:48.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:48.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:48.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:48.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:48.375+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:48.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:48.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:48.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:48.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:48.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:48.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:48.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:48.527+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:49.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:48.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:49.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:48.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:49.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:48.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:49.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:48.731+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:49.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:49.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:49.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:49.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:49.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:49.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:49.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:49.399+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:49.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:49.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:49.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:49.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:49.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:49.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:49.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:49.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:50.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:49.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:50.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:49.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:50.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:49.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:50.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:49.754+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:50.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:50.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:50.547 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:50.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:50.547 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:50.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:50.547 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:50.409+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:51.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:50.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:51.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:50.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:51.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:50.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:51.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:50.590+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:51.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:50.739+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:51.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:50.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:51.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:50.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:51.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:50.740+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:51.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:51.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:51.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:51.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:51.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:51.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:51.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:51.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:51.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:51.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:51.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:51.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:51.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:51.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:51.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:51.585+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:52.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:51.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:52.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:51.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:52.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:51.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:52.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:51.697+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:52.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:52.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:52.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:52.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:52.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:52.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:52.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:52.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:52.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:52.449+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:52.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:52.449+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:52.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:52.449+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:52.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:52.450+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:52.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:52.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:52.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:52.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:52.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:52.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:52.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:52.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:53.796 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:53.417+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21213 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:53.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:53.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:53.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:53.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:53.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:53.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:53.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:53.423+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:53.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:53.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:53.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:53.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:53.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:53.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:53.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:53.544+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:53.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:53.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:53.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:53.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:53.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:53.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:53.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:53.665+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:54.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:54.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:54.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:54.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:54.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:54.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:54.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:54.405+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:54.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:54.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:54.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:54.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:54.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:54.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:54.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:54.551+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:54.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:54.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:54.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:54.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:54.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:54.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:54.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:54.646+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:55.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:55.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:55.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:55.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:55.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:55.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:55.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:55.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:55.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:55.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:55.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:55.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:55.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:55.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:55.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:55.585+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:55.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:55.636+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:55.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:55.636+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:55.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:55.636+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:55.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:55.636+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:56.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:56.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:56.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:56.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:56.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:56.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:56.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:56.400+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:56.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:56.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:56.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:56.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:56.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:56.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:56.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:56.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:57.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:56.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:57.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:56.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:57.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:56.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:57.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:56.673+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:57.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:57.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:57.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:57.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:57.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:57.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:57.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:57.549+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:57.728 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:57.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:57.728 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:57.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:57.728 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:57.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:57.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:57.449+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:58.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:57.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:58.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:57.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:58.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:57.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:58.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:57.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:58.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:26:58.421+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21296 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:26:58.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:58.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:58.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:58.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:58.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:58.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:58.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:58.482+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:58.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:58.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:58.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:58.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:58.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:58.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:58.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:58.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:59.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:58.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:59.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:58.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:59.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:58.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:59.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:58.714+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:59.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:59.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:59.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:26:59.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:26:59.610+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:26:59.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:59.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:59.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:59.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:59.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:59.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:26:59.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:26:59.489+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:26:59.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:59.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:59.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:59.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:59.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:59.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:26:59.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:26:59.731+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:00.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:00.739+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:00.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:00.739+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:00.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:00.739+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:00.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:00.739+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:00.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:00.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:00.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:00.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:00.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:00.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:00.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:00.639+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:00.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:00.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:00.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:00.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:00.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:00.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:00.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:00.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:02.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:01.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:02.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:01.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:02.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:01.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:02.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:01.708+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:02.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:01.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:02.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:01.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:02.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:01.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:02.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:01.619+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:02.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:01.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:02.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:01.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:02.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:01.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:02.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:01.549+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:02.870 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:02.579+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:02.870 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:02.579+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:02.870 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:02.579+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:02.870 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:02.579+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:02.871 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:02.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:02.871 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:02.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:02.871 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:02.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:02.871 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:02.568+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:02.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:02.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:02.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:02.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:02.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:02.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:02.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:02.721+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:03.731 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:03.424+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21376 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:03.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:03.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:03.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:03.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:03.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:03.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:03.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:03.563+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:03.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:03.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:03.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:03.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:03.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:03.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:03.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:03.567+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:04.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:03.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:04.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:03.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:04.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:03.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:04.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:03.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:05.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:04.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:05.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:04.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:05.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:04.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:05.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:04.558+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:05.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:04.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:05.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:04.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:05.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:04.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:05.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:04.544+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:05.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:04.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:05.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:04.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:05.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:04.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:05.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:04.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:05.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:05.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:05.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:05.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:05.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:05.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:05.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:05.720+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:05.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:05.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:05.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:05.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:05.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:05.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:05.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:05.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:05.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:05.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:05.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:05.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:05.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:05.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:05.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:05.528+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:07.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:06.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:07.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:06.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:07.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:06.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:07.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:06.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:07.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:06.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:07.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:06.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:07.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:06.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:07.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:06.620+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:07.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:06.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:07.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:06.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:07.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:06.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:07.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:06.550+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:08.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:07.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:08.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:07.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:08.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:07.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:08.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:07.720+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:08.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:07.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:08.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:07.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:08.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:07.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:08.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:07.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:08.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:07.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:08.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:07.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:08.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:07.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:08.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:07.574+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:08.796 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:08.428+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21456 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:08.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:08.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:08.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:08.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:08.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:08.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:08.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:08.676+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:08.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:08.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:08.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:08.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:08.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:08.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:08.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:08.646+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:08.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:08.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:08.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:08.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:08.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:08.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:08.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:08.569+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:09.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:09.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:09.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:09.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:09.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:09.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:09.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:09.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:09.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:09.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:09.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:09.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:09.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:09.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:09.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:09.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:09.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:09.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:09.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:09.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:09.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:09.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:09.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:09.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:11.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:10.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:11.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:10.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:11.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:10.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:11.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:10.670+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:11.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:10.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:11.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:10.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:11.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:10.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:11.048 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:10.696+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:11.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:10.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:11.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:10.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:11.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:10.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:11.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:10.544+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:11.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:11.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:11.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:11.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:11.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:11.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:11.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:11.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:11.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:11.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:11.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:11.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:11.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:11.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:11.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:11.504+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:11.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:11.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:11.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:11.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:11.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:11.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:11.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:11.654+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:12.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:12.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:12.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:12.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:12.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:12.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:12.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:12.740+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:12.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:12.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:12.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:12.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:12.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:12.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:12.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:12.679+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:12.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:12.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:12.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:12.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:12.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:12.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:12.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:12.530+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:13.708 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:13.431+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21537 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:13.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:13.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:13.709 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:13.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:13.709 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:13.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:13.709 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:13.524+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:14.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:13.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:14.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:13.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:14.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:13.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:14.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:13.703+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:14.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:13.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:14.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:13.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:14.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:13.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:14.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:13.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:14.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:14.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:14.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:14.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:14.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:14.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:14.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:14.660+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:14.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:14.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:14.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:14.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:14.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:14.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:14.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:14.480+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:15.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:14.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:15.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:14.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:15.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:14.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:15.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:14.733+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:15.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:15.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:15.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:15.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:15.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:15.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:15.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:15.481+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:15.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:15.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:15.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:15.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:15.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:15.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:15.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:15.662+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:15.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:15.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:15.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:15.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:15.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:15.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:15.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:15.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:16.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:16.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:16.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:16.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:16.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:16.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:16.798 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:16.449+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:16.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:16.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:16.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:16.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:16.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:16.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:16.670+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:16.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:16.799 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:16.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:16.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:16.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:16.800 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:16.683+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:17.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:17.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:17.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:17.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:17.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:17.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:17.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:17.667+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:17.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:17.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:17.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:17.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:17.716 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:17.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:17.716 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:17.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:18.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:17.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:18.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:17.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:18.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:17.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:18.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:17.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:18.684 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:18.435+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21623 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:18.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:18.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:18.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:18.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:18.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:18.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:18.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:18.405+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:19.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:18.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:19.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:18.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:19.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:18.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:19.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:18.738+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:19.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:18.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:19.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:18.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:19.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:18.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:19.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:18.679+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:19.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:19.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:19.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:19.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:19.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:19.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:19.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:19.424+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:20.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:19.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:20.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:19.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:20.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:19.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:20.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:19.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:20.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:19.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:20.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:19.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:20.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:19.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:20.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:19.691+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:20.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:20.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:20.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:20.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:20.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:20.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:20.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:20.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:20.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:20.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:20.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:20.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:20.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:20.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:20.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:20.456+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:20.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:20.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:20.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:20.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:20.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:20.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:20.798 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:20.668+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:21.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:21.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:21.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:21.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:21.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:21.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:21.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:21.474+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:22.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:21.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:22.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:21.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:22.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:21.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:22.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:21.685+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:22.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:21.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:22.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:21.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:22.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:21.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:22.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:21.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:22.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:22.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:22.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:22.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:22.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:22.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:22.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:22.454+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:23.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:22.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:23.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:22.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:23.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:22.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:23.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:22.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:23.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:22.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:23.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:22.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:23.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:22.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:23.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:22.727+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:23.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:23.438+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21712 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:23.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:23.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:23.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:23.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:23.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:23.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:23.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:23.418+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:24.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:23.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:24.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:23.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:24.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:23.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:24.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:23.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:24.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:23.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:24.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:23.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:24.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:23.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:24.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:23.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:24.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:24.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:24.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:24.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:24.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:24.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:24.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:24.409+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:25.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:24.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:25.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:24.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:25.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:24.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:25.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:24.717+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:25.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:24.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:25.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:24.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:25.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:24.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:25.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:24.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:25.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:25.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:25.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:25.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:25.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:25.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:25.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:25.380+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:26.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:25.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:26.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:25.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:26.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:25.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:26.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:25.702+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:26.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:25.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:26.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:25.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:26.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:25.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:26.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:25.694+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:26.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:26.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:26.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:26.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:26.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:26.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:26.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:26.425+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:27.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:26.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:27.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:26.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:27.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:26.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:27.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:26.745+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:27.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:26.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:27.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:26.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:27.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:26.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:27.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:26.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:27.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:27.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:27.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:27.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:27.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:27.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:27.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:27.668+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:27.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:27.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:27.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:27.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:27.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:27.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:27.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:27.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:28.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:27.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:28.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:27.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:28.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:27.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:28.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:27.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:28.775 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:28.443+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21801 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:28.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:28.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:28.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:28.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:28.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:28.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:28.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:28.428+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:28.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:28.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:28.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:28.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:28.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:28.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:28.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:28.667+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:29.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:28.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:29.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:28.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:29.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:28.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:29.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:28.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:29.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:29.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:29.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:29.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:29.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:29.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:29.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:29.462+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:29.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:29.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:29.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:29.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:29.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:29.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:29.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:29.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:30.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:29.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:30.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:29.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:30.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:29.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:30.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:29.777+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:30.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:30.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:30.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:30.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:30.471+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:30.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:30.703+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:30.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:30.703+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:30.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:30.703+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:30.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:30.703+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:31.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:30.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:31.298 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:30.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:31.298 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:30.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:31.298 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:30.818+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:31.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:31.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:31.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:31.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:31.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:31.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:31.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:31.500+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:31.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:31.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:31.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:31.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:31.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:31.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:31.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:31.735+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:32.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:31.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:32.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:31.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:32.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:31.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:32.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:31.852+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:32.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:32.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:32.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:32.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:32.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:32.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:32.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:32.452+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:32.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:32.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:32.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:32.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:32.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:32.687+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:32.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:32.688+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:33.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:32.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:33.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:32.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:33.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:32.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:33.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:32.848+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:33.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:33.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:33.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:33.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:33.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:33.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:33.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:33.405+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:33.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:33.445+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21891 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:34.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:33.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:34.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:33.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:34.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:33.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:34.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:33.805+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:34.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:33.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:34.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:33.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:34.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:33.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:34.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:33.736+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:34.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:34.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:34.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:34.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:34.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:34.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:34.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:34.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:35.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:34.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:35.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:34.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:35.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:34.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:35.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:34.813+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:35.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:34.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:35.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:34.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:35.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:34.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:35.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:34.738+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:35.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:35.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:35.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:35.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:35.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:35.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:35.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:35.339+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:36.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:35.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:36.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:35.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:36.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:35.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:36.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:35.830+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:36.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:35.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:36.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:35.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:36.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:35.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:36.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:35.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:36.662 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:36.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:36.662 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:36.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:36.662 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:36.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:36.662 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:36.322+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:37.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:36.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:36.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:36.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:37.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:36.855+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:37.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:36.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:37.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:36.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:37.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:36.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:37.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:36.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:37.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:37.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:37.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:37.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:37.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:37.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:37.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:37.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:38.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:37.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:38.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:37.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:38.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:37.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:38.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:37.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:38.297 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:37.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:38.297 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:37.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:38.297 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:37.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:38.297 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:37.803+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:38.579 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:38.449+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 21980 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:38.579 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:38.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:38.579 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:38.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:38.579 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:38.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:38.579 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:38.293+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:39.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:38.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:39.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:38.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:39.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:38.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:39.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:38.755+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:39.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:38.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:39.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:38.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:39.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:38.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:39.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:38.807+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:39.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:39.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:39.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:39.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:39.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:39.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:39.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:39.274+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:40.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:39.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:40.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:39.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:40.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:39.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:40.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:39.787+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:40.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:39.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:40.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:39.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:40.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:39.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:40.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:39.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:40.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:40.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:40.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:40.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:40.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:40.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:40.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:40.259+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:41.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:40.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:41.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:40.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:41.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:40.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:41.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:40.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:41.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:40.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:41.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:40.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:41.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:40.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:41.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:40.742+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:41.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:41.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:41.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:41.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:41.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:41.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:41.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:41.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:42.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:41.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:42.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:41.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:42.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:41.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:42.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:41.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:42.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:41.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:42.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:41.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:42.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:41.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:42.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:41.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:42.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:42.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:42.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:42.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:42.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:42.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:42.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:42.264+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:43.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:42.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:43.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:42.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:43.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:42.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:43.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:42.799+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:42.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:42.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:42.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:43.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:42.772+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:43.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:43.453+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22069 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:43.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:43.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:43.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:43.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:43.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:43.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:43.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:43.284+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:44.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:43.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:44.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:43.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:44.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:43.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:44.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:43.801+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:44.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:43.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:44.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:43.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:44.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:43.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:44.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:43.773+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:44.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:44.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:44.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:44.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:44.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:44.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:44.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:44.305+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:45.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:44.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:45.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:44.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:45.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:44.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:45.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:44.803+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:45.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:44.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:45.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:44.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:45.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:44.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:45.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:44.770+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:45.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:45.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:45.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:45.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:45.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:45.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:45.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:45.318+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:46.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:45.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:46.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:45.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:46.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:45.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:46.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:45.792+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:46.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:45.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:46.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:45.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:46.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:45.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:46.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:45.802+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:46.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:46.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:46.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:46.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:46.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:46.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:46.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:46.308+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:47.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:46.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:47.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:46.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:47.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:46.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:47.297 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:46.807+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:47.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:46.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:47.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:46.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:47.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:46.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:47.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:46.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:47.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:47.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:47.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:47.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:47.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:47.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:47.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:47.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:48.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:47.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:48.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:47.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:48.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:47.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:48.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:47.801+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:48.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:47.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:48.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:47.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:48.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:47.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:48.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:47.766+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:48.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:48.456+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22158 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:48.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:48.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:48.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:48.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:48.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:48.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:48.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:48.354+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:49.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:48.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:49.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:48.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:49.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:48.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:49.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:48.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:49.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:48.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:49.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:48.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:49.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:48.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:49.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:48.720+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:49.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:49.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:49.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:49.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:49.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:49.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:49.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:49.336+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:50.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:49.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:50.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:49.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:50.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:49.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:50.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:49.751+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:50.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:49.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:50.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:49.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:50.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:49.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:50.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:49.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:50.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:50.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:50.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:50.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:50.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:50.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:50.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:50.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:51.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:50.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:51.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:50.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:51.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:50.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:51.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:50.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:51.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:50.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:51.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:50.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:51.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:50.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:51.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:50.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:51.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:51.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:51.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:51.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:51.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:51.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:51.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:51.343+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:52.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:51.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:52.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:51.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:52.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:51.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:52.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:51.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:52.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:51.675+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:52.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:51.675+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:52.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:51.675+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:52.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:51.676+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:52.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:52.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:52.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:52.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:52.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:52.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:52.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:52.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:53.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:52.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:53.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:52.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:53.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:52.763+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:53.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:52.764+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:53.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:52.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:53.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:52.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:53.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:52.654+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:53.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:52.654+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:53.691 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:53.460+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22248 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:53.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:53.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:53.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:53.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:53.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:53.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:53.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:53.373+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:54.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:53.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:54.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:53.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:54.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:53.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:54.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:53.799+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:54.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:53.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:54.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:53.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:54.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:53.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:54.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:53.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:54.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:54.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:54.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:54.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:54.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:54.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:54.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:54.365+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:55.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:54.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:55.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:54.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:55.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:54.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:55.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:54.786+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:55.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:54.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:55.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:54.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:55.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:54.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:55.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:54.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:55.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:55.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:55.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:55.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:55.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:55.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:55.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:55.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:56.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:55.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:56.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:55.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:56.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:55.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:56.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:55.824+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:56.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:55.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:56.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:55.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:56.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:55.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:56.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:55.752+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:56.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:56.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:56.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:56.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:56.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:56.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:56.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:56.324+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:57.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:56.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:57.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:56.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:57.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:56.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:57.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:56.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:57.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:56.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:57.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:56.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:57.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:56.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:57.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:56.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:57.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:57.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:57.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:57.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:57.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:57.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:57.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:57.276+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:58.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:57.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:58.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:57.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:58.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:57.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:58.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:57.767+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:58.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:57.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:58.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:57.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:58.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:57.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:58.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:57.743+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:58.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:27:58.463+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22337 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:27:58.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:58.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:58.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:58.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:58.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:58.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:58.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:58.257+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:27:59.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:58.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:59.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:58.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:59.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:58.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:27:59.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:58.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:59.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:58.783+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:59.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:58.783+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:59.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:58.783+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:27:59.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:58.783+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:27:59.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:59.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:59.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:59.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:59.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:59.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:27:59.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:27:59.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:00.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:59.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:00.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:59.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:00.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:59.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:00.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:27:59.724+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:00.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:59.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:00.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:59.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:00.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:59.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:00.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:27:59.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:00.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:00.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:00.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:00.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:00.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:00.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:00.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:00.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:01.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:00.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:01.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:00.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:01.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:00.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:01.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:00.729+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:01.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:00.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:01.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:00.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:01.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:00.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:01.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:00.712+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:01.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:01.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:01.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:01.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:01.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:01.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:01.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:01.208+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:02.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:01.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:02.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:01.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:02.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:01.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:02.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:01.702+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:02.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:01.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:02.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:01.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:02.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:01.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:02.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:01.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:02.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:02.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:02.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:02.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:02.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:02.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:02.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:02.159+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:03.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:02.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:03.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:02.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:03.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:02.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:03.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:02.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:03.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:02.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:03.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:02.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:03.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:02.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:03.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:02.682+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:03.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:03.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:03.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:03.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:03.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:03.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:03.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:03.176+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:03.770 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:03.467+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22426 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:03.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:03.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:03.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:03.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:03.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:03.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:03.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:03.647+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:04.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:03.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:04.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:03.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:04.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:03.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:04.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:03.766+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:04.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:04.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:04.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:04.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:04.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:04.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:04.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:04.190+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:05.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:04.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:05.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:04.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:05.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:04.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:05.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:04.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:05.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:04.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:05.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:04.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:05.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:04.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:05.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:04.605+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:05.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:05.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:05.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:05.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:05.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:05.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:05.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:05.222+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:06.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:05.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:06.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:05.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:06.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:05.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:06.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:05.782+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:06.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:05.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:06.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:05.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:06.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:05.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:06.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:05.580+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:06.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:06.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:06.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:06.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:06.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:06.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:06.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:06.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:07.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:06.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:07.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:06.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:07.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:06.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:07.047 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:06.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:07.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:06.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:07.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:06.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:07.048 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:06.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:07.049 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:06.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:07.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:07.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:07.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:07.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:07.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:07.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:07.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:07.272+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:07.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:07.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:07.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:07.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:07.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:07.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:07.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:07.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:08.262 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:07.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:08.262 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:07.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:08.262 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:07.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:08.262 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:07.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:08.546 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:08.471+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22515 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:08.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:08.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:08.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:08.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:08.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:08.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:08.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:08.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:08.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:08.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:08.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:08.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:08.548 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:08.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:08.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:08.485+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:09.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:08.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:09.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:08.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:09.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:08.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:09.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:08.758+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:09.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:09.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:09.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:09.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:09.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:09.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:09.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:09.237+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:09.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:09.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:09.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:09.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:09.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:09.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:09.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:09.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:10.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:09.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:10.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:09.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:10.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:09.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:10.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:09.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:10.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:10.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:10.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:10.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:10.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:10.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:10.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:10.286+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:10.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:10.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:10.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:10.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:10.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:10.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:10.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:10.560+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:11.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:10.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:11.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:10.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:11.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:10.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:11.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:10.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:11.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:11.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:11.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:11.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:11.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:11.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:11.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:11.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:11.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:11.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:11.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:11.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:11.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:11.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:11.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:11.294+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:12.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:11.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:12.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:11.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:12.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:11.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:12.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:11.783+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:12.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:12.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:12.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:12.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:12.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:12.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:12.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:12.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:12.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:12.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:12.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:12.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:12.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:12.330+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:12.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:12.331+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:12.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:12.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:12.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:12.783+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:13.602 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:13.475+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22605 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:13.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:13.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:13.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:13.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:13.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:13.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:13.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:13.320+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:13.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:13.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:13.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:13.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:13.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:13.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:13.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:13.485+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:14.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:13.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:14.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:13.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:14.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:13.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:14.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:13.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:14.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:14.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:14.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:14.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:14.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:14.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:14.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:14.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:14.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:14.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:14.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:14.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:14.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:14.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:14.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:14.336+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:15.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:14.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:15.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:14.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:15.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:14.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:15.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:14.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:15.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:15.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:15.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:15.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:15.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:15.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:15.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:15.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:15.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:15.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:15.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:15.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:15.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:15.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:15.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:15.308+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:16.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:15.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:16.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:15.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:16.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:15.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:16.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:15.826+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:16.597 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:16.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:16.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:16.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:16.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:16.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:16.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:16.322+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:16.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:16.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:16.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:16.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:16.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:16.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:16.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:16.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:17.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:16.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:17.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:16.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:17.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:16.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:17.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:16.875+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:17.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:17.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:17.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:17.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:17.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:17.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:17.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:17.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:17.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:17.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:17.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:17.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:17.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:17.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:17.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:17.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:18.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:17.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:18.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:17.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:18.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:17.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:18.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:17.899+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:18.796 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:18.479+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22694 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:18.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:18.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:18.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:18.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:18.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:18.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:18.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:18.362+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:18.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:18.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:18.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:18.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:18.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:18.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:18.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:18.462+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:19.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:18.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:19.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:18.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:19.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:18.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:19.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:18.908+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:19.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:19.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:19.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:19.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:19.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:19.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:19.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:19.351+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:19.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:19.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:19.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:19.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:19.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:19.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:19.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:19.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:20.189 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:19.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:20.189 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:19.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:20.189 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:19.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:20.189 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:19.914+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:20.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:20.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:20.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:20.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:20.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:20.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:20.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:20.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:20.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:20.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:20.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:20.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:20.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:20.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:20.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:20.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:21.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:20.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:21.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:20.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:21.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:20.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:21.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:20.957+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:21.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:21.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:21.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:21.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:21.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:21.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:21.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:21.302+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:21.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:21.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:21.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:21.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:21.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:21.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:21.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:21.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:22.274 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:21.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:22.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:21.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:22.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:21.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:22.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:21.938+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:22.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:22.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:22.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:22.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:22.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:22.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:22.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:22.270+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:22.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:22.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:22.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:22.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:22.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:22.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:22.547 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:22.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:23.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:22.967+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:23.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:22.967+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:23.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:22.967+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:23.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:22.967+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:23.614 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:23.483+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22783 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:23.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:23.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:23.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:23.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:23.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:23.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:23.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:23.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:23.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:23.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:23.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:23.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:23.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:23.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:23.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:23.320+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:24.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:23.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:24.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:23.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:24.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:23.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:24.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:23.978+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:24.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:24.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:24.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:24.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:24.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:24.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:24.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:24.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:24.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:24.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:24.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:24.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:24.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:24.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:24.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:24.278+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:25.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:24.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:25.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:24.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:25.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:24.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:25.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:24.938+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:25.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:25.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:25.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:25.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:25.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:25.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:25.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:25.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:25.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:25.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:25.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:25.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:25.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:25.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:25.547 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:25.273+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:26.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:25.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:26.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:25.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:26.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:25.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:26.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:25.924+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:26.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:26.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:26.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:26.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:26.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:26.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:26.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:26.404+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:26.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:26.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:26.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:26.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:26.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:26.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:26.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:26.298+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:27.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:26.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:27.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:26.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:27.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:26.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:27.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:26.932+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:27.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:27.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:27.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:27.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:27.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:27.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:27.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:27.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:27.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:27.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:27.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:27.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:27.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:27.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:27.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:27.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:28.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:27.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:28.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:27.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:28.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:27.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:28.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:27.970+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:28.796 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:28.487+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22872 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:28.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:28.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:28.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:28.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:28.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:28.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:28.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:28.384+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:28.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:28.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:28.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:28.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:28.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:28.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:28.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:28.385+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:29.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:28.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:29.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:28.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:29.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:28.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:29.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:28.924+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:29.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:29.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:29.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:29.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:29.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:29.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:29.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:29.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:29.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:29.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:29.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:29.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:29.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:29.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:29.384+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:30.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:29.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:30.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:29.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:30.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:29.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:30.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:29.941+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:30.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:30.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:30.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:30.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:30.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:30.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:30.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:30.411+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:30.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:30.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:30.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:30.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:30.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:30.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:30.376+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:30.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:31.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:30.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:31.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:30.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:31.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:30.950+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:31.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:31.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:31.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:31.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:31.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:31.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:31.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:31.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:31.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:31.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:31.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:31.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:31.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:31.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:31.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:31.416+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:32.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:32.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:32.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:32.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:32.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:32.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:32.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:32.001+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:32.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:32.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:32.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:32.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:32.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:32.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:32.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:32.457+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:32.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:32.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:32.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:32.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:32.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:32.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:32.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:32.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:33.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:32.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:33.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:32.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:33.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:32.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:33.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:32.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:33.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:33.490+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 22962 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:33.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:33.437+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:33.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:33.437+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:33.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:33.437+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:33.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:33.438+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:33.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:33.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:33.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:33.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:33.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:33.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:33.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:33.455+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:34.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:33.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:34.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:33.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:34.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:33.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:34.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:33.948+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:34.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:34.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:34.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:34.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:34.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:34.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:34.457+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:34.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:34.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:34.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:34.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:34.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:34.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:35.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:34.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:35.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:34.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:35.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:34.951+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:35.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:35.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:35.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:35.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:35.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:35.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:35.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:35.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:35.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:35.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:35.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:35.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:35.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:35.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:35.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:35.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:36.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:35.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:36.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:35.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:36.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:35.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:36.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:35.990+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:36.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:36.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:36.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:36.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:36.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:36.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:36.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:36.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:36.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:36.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:36.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:36.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:36.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:36.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:36.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:36.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:37.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:36.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:37.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:36.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:37.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:36.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:37.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:36.975+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:37.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:37.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:37.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:37.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:37.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:37.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:37.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:37.476+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:37.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:37.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:37.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:37.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:37.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:37.455+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:38.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:37.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:38.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:37.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:38.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:37.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:38.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:37.960+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:38.623 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:38.494+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23051 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:38.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:38.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:38.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:38.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:38.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:38.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:38.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:38.506+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:38.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:38.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:38.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:38.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:38.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:38.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:38.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:38.476+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:39.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:39.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:39.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:39.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:39.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:39.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:39.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:39.010+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:39.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:39.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:39.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:39.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:39.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:39.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:39.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:39.510+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:39.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:39.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:39.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:39.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:39.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:39.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:39.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:39.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:40.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:40.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:40.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:40.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:40.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:40.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:40.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:40.020+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:40.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:40.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:40.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:40.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:40.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:40.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:40.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:40.557+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:40.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:40.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:40.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:40.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:40.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:40.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:40.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:40.475+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:41.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:41.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:41.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:41.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:41.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:41.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:41.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:41.064+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:41.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:41.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:41.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:41.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:41.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:41.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:41.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:41.581+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:41.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:41.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:41.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:41.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:41.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:41.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:41.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:41.451+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:42.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:42.046+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:42.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:42.046+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:42.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:42.046+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:42.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:42.046+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:42.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:42.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:42.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:42.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:42.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:42.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:42.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:42.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:42.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:42.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:42.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:42.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:42.460+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:43.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:43.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:43.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:43.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:43.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:43.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:43.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:43.024+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:43.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:43.498+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23140 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:43.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:43.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:43.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:43.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:43.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:43.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:43.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:43.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:43.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:43.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:43.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:43.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:43.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:43.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:43.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:43.474+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:44.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:44.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:44.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:44.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:44.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:44.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:44.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:44.004+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:44.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:44.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:44.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:44.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:44.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:44.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:44.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:44.503+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:44.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:44.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:44.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:44.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:44.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:44.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:44.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:44.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:45.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:44.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:45.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:44.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:45.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:44.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:45.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:44.979+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:45.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:45.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:45.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:45.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:45.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:45.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:45.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:45.526+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:45.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:45.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:45.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:45.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:45.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:45.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:45.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:45.597+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:46.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:45.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:46.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:45.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:46.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:45.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:46.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:45.945+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:46.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:46.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:46.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:46.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:46.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:46.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:46.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:46.544+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:46.959 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:46.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:46.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:46.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:46.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:46.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:46.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:46.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:47.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:46.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:47.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:46.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:47.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:46.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:47.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:46.955+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:47.960 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:47.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:47.960 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:47.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:47.960 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:47.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:47.960 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:47.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:47.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:47.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:47.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:47.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:47.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:47.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:47.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:47.579+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:47.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:47.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:47.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:47.956+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:48.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:48.502+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23229 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:48.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:48.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:48.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:48.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:48.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:48.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:48.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:48.522+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:48.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:48.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:48.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:48.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:48.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:48.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:48.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:48.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:49.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:48.939+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:49.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:48.939+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:49.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:48.939+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:49.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:48.939+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:49.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:49.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:49.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:49.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:49.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:49.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:49.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:49.565+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:49.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:49.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:49.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:49.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:49.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:49.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:49.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:49.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:50.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:49.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:50.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:49.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:50.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:49.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:50.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:49.966+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:50.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:50.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:50.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:50.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:50.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:50.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:50.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:50.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:50.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:50.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:50.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:50.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:50.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:50.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:50.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:50.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:51.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:50.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:51.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:50.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:51.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:50.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:51.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:50.947+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:51.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:51.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:51.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:51.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:51.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:51.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:51.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:51.470+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:51.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:51.539+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:51.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:51.539+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:51.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:51.539+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:51.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:51.540+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:52.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:51.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:52.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:51.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:52.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:51.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:52.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:51.912+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:52.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:52.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:52.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:52.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:52.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:52.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:52.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:52.529+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:52.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:52.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:52.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:52.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:52.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:52.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:52.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:52.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:53.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:52.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:53.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:52.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:53.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:52.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:53.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:52.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:53.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:53.505+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23319 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:53.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:53.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:53.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:53.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:53.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:53.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:53.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:53.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:53.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:53.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:53.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:53.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:53.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:53.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:53.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:53.510+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:54.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:53.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:54.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:53.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:54.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:53.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:54.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:53.886+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:54.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:54.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:54.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:54.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:54.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:54.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:54.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:54.475+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:54.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:54.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:54.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:54.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:54.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:54.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:54.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:54.401+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:55.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:54.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:55.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:54.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:55.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:54.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:55.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:54.890+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:55.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:55.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:55.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:55.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:55.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:55.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:55.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:55.427+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:55.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:55.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:55.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:55.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:55.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:55.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:55.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:55.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:56.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:55.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:56.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:55.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:56.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:55.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:56.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:55.842+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:56.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:56.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:56.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:56.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:56.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:56.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:56.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:56.428+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:56.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:56.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:56.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:56.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:56.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:56.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:56.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:56.339+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:57.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:56.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:57.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:56.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:57.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:56.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:57.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:56.844+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:57.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:57.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:57.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:57.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:57.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:57.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:57.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:57.424+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:57.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:57.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:57.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:57.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:57.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:57.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:57.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:57.363+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:58.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:57.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:58.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:57.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:58.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:57.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:58.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:57.855+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:58.638 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:28:58.509+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23408 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:28:58.639 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:58.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:58.639 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:58.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:58.639 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:58.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:58.639 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:58.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:58.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:58.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:58.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:58.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:58.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:58.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:58.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:58.330+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:59.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:58.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:59.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:58.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:59.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:58.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:28:59.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:58.897+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:28:59.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:59.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:59.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:59.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:59.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:59.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:28:59.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:28:59.416+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:28:59.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:59.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:59.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:59.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:59.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:59.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:28:59.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:28:59.372+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:00.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:59.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:00.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:59.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:00.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:59.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:00.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:28:59.874+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:00.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:00.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:00.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:00.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:00.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:00.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:00.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:00.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:00.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:00.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:00.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:00.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:00.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:00.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:00.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:00.419+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:01.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:00.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:01.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:00.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:01.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:00.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:01.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:00.904+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:01.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:01.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:01.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:01.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:01.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:01.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:01.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:01.364+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:01.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:01.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:01.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:01.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:01.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:01.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:01.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:01.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:02.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:01.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:02.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:01.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:02.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:01.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:02.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:01.908+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:02.677 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:02.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:02.677 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:02.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:02.677 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:02.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:02.677 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:02.323+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:02.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:02.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:02.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:02.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:02.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:02.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:02.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:02.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:03.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:02.935+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:03.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:02.935+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:03.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:02.935+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:03.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:02.936+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:03.616 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:03.513+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23497 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:03.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:03.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:03.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:03.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:03.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:03.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:03.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:03.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:03.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:03.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:03.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:03.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:03.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:03.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:03.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:03.315+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:04.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:03.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:04.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:03.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:04.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:03.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:04.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:03.897+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:04.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:04.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:04.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:04.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:04.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:04.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:04.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:04.321+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:04.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:04.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:04.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:04.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:04.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:04.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:04.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:04.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:05.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:04.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:05.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:04.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:05.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:04.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:05.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:04.938+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:05.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:05.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:05.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:05.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:05.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:05.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:05.546 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:05.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:05.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:05.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:05.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:05.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:05.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:05.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:05.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:05.282+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:06.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:05.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:06.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:05.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:06.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:05.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:06.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:05.978+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:06.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:06.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:06.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:06.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:06.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:06.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:06.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:06.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:06.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:06.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:06.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:06.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:06.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:06.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:06.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:06.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:07.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:06.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:07.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:06.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:07.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:06.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:07.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:06.947+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:07.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:07.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:07.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:07.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:07.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:07.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:07.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:07.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:07.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:07.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:07.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:07.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:07.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:07.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:07.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:07.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:08.192 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:07.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:08.192 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:07.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:08.192 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:07.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:08.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:07.915+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:08.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:08.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:08.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:08.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:08.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:08.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:08.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:08.188+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:08.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:08.517+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23586 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:08.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:08.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:08.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:08.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:08.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:08.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:08.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:08.531+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:09.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:08.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:09.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:08.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:09.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:08.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:09.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:08.896+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:09.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:09.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:09.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:09.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:09.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:09.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:09.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:09.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:09.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:09.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:09.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:09.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:09.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:09.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:09.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:09.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:10.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:09.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:10.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:09.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:10.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:09.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:10.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:09.877+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:10.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:10.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:10.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:10.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:10.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:10.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:10.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:10.191+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:10.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:10.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:10.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:10.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:10.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:11.184 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:10.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:11.184 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:10.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:11.184 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:10.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:11.185 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:10.928+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:11.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:11.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:11.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:11.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:11.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:11.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:11.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:11.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:11.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:11.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:11.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:11.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:11.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:11.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:11.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:11.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:12.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:11.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:12.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:11.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:12.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:11.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:12.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:11.897+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:12.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:12.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:12.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:12.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:12.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:12.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:12.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:12.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:12.853 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:12.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:12.853 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:12.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:12.853 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:12.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:12.853 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:12.575+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:12.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:12.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:12.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:12.849+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:13.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:13.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:13.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:13.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:13.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:13.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:13.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:13.218+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:13.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:13.521+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23676 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:13.818 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:13.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:13.819 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:13.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:13.819 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:13.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:13.819 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:13.592+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:14.223 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:13.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:14.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:13.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:14.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:13.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:14.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:13.892+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:14.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:14.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:14.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:14.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:14.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:14.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:14.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:14.220+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:14.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:14.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:14.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:14.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:14.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:14.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:14.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:14.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:15.186 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:14.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:15.186 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:14.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:15.186 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:14.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:15.186 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:14.892+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:15.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:15.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:15.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:15.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:15.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:15.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:15.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:15.182+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:15.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:15.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:15.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:15.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:15.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:15.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:15.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:15.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:16.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:15.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:16.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:15.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:16.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:15.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:16.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:15.908+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:16.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:16.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:16.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:16.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:16.297 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:16.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:16.297 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:16.143+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:16.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:16.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:16.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:16.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:16.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:16.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:16.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:16.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:17.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:16.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:17.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:16.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:17.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:16.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:17.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:16.931+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:17.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:17.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:17.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:17.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:17.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:17.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:17.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:17.158+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:17.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:17.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:17.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:17.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:17.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:17.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:17.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:17.619+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:18.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:17.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:18.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:17.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:18.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:17.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:18.199 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:17.890+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:18.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:18.195+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:18.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:18.195+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:18.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:18.195+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:18.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:18.195+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:18.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:18.525+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23765 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:18.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:18.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:18.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:18.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:18.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:18.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:18.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:18.660+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:19.238 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:18.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:19.238 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:18.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:19.238 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:18.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:19.238 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:18.855+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:19.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:19.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:19.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:19.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:19.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:19.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:19.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:19.234+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:19.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:19.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:19.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:19.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:19.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:19.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:19.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:19.646+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:20.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:19.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:20.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:19.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:20.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:19.866+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:20.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:19.867+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:20.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:20.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:20.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:20.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:20.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:20.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:20.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:20.256+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:21.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:20.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:21.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:20.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:21.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:20.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:21.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:20.661+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:21.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:20.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:21.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:20.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:21.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:20.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:21.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:20.909+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:21.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:21.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:21.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:21.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:21.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:21.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:21.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:21.302+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:21.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:21.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:21.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:21.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:21.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:21.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:21.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:21.624+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:22.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:21.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:22.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:21.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:22.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:21.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:22.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:21.914+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:22.614 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:22.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:22.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:22.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:22.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:22.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:22.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:22.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:22.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:22.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:22.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:22.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:22.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:22.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:22.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:22.610+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:23.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:22.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:23.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:22.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:23.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:22.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:23.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:22.878+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:23.617 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:23.529+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23854 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:23.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:23.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:23.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:23.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:23.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:23.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:23.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:23.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:23.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:23.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:23.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:23.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:23.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:23.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:23.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:23.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:24.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:23.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:24.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:23.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:24.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:23.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:24.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:23.875+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:24.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:24.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:24.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:24.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:24.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:24.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:24.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:24.354+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:24.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:24.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:24.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:24.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:24.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:24.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:24.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:24.606+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:25.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:24.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:25.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:24.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:25.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:24.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:25.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:24.911+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:25.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:25.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:25.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:25.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:25.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:25.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:25.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:25.403+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:25.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:25.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:25.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:25.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:25.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:25.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:25.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:25.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:26.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:25.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:26.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:25.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:26.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:25.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:26.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:25.924+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:26.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:26.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:26.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:26.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:26.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:26.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:26.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:26.426+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:27.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:26.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:27.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:26.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:27.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:26.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:27.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:26.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:27.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:26.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:27.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:26.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:27.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:26.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:27.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:26.680+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:27.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:27.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:27.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:27.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:27.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:27.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:27.705 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:27.430+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:28.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:27.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:28.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:27.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:28.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:27.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:28.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:27.911+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:28.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:27.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:28.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:27.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:28.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:27.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:28.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:27.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:28.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:28.533+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 23943 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:28.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:28.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:28.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:28.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:28.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:28.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:28.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:28.422+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:29.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:28.930+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:29.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:28.930+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:29.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:28.930+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:29.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:28.930+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:29.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:28.748+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:29.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:28.748+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:29.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:28.748+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:29.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:28.748+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:29.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:29.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:29.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:29.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:29.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:29.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:29.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:29.422+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:30.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:29.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:30.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:29.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:30.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:29.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:30.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:29.702+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:30.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:29.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:30.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:29.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:30.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:29.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:30.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:29.901+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:30.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:30.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:30.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:30.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:30.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:30.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:30.455+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:30.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:30.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:30.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:30.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:30.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:30.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:30.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:30.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:30.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:30.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:31.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:30.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:31.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:30.863+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:31.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:31.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:31.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:31.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:31.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:31.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:31.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:31.448+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:31.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:31.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:31.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:31.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:31.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:31.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:31.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:31.688+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:32.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:31.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:32.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:31.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:32.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:31.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:32.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:31.873+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:32.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:32.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:32.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:32.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:32.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:32.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:32.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:32.469+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:32.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:32.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:32.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:32.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:32.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:32.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:32.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:32.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:33.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:32.893+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:33.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:32.893+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:33.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:32.893+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:33.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:32.893+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:33.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:33.538+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24032 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:33.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:33.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:33.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:33.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:33.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:33.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:33.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:33.487+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:33.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:33.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:33.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:33.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:33.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:33.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:33.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:33.636+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:34.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:33.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:34.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:33.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:34.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:33.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:34.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:33.941+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:34.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:34.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:34.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:34.510+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:34.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:34.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:34.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:34.664+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:34.980+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:34.980+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:34.980+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:35.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:34.981+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:35.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:35.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:35.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:35.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:35.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:35.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:35.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:35.470+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:35.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:35.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:35.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:35.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:35.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:35.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:35.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:35.630+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:36.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:35.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:36.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:35.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:36.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:35.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:36.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:35.954+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:36.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:36.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:36.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:36.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:36.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:36.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:36.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:36.468+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:36.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:36.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:36.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:36.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:36.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:36.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:36.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:36.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:37.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:36.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:37.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:36.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:37.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:36.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:37.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:36.965+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:37.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:37.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:37.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:37.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:37.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:37.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:37.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:37.437+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:37.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:37.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:37.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:37.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:37.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:37.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:37.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:37.606+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:38.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:38.014+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:38.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:38.014+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:38.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:38.014+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:38.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:38.015+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:38.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:38.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:38.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:38.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:38.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:38.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:38.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:38.460+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:38.990 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:38.541+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24123 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:38.990 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:38.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:38.991 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:38.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:38.991 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:38.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:38.991 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:38.567+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:39.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:38.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:39.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:38.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:39.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:38.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:39.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:38.986+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:39.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:39.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:39.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:39.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:39.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:39.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:39.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:39.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:39.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:39.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:39.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:39.477+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:40.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:39.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:40.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:39.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:40.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:39.956+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:40.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:39.957+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:40.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:40.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:40.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:40.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:40.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:40.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:40.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:40.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:40.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:40.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:40.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:40.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:40.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:40.447+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:40.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:40.447+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:41.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:40.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:41.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:40.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:41.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:40.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:41.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:40.916+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:41.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:41.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:41.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:41.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:41.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:41.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:41.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:41.407+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:41.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:41.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:41.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:41.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:41.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:41.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:41.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:41.467+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:42.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:41.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:42.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:41.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:42.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:41.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:42.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:41.960+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:42.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:42.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:42.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:42.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:42.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:42.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:42.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:42.421+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:42.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:42.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:42.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:42.479+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:43.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:42.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:43.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:42.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:43.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:42.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:43.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:42.942+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:43.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:43.545+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24212 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:43.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:43.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:43.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:43.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:43.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:43.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:43.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:43.462+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:43.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:43.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:43.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:43.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:43.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:43.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:43.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:43.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:44.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:43.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:44.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:43.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:44.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:43.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:44.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:43.958+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:44.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:44.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:44.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:44.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:44.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:44.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:44.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:44.443+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:44.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:44.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:44.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:44.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:44.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:44.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:44.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:44.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:45.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:44.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:45.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:44.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:45.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:44.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:45.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:44.953+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:45.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:45.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:45.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:45.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:45.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:45.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:45.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:45.455+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:45.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:45.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:45.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:45.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:45.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:45.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:45.686 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:45.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:46.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:45.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:46.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:45.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:46.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:45.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:46.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:45.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:46.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:46.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:46.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:46.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:46.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:46.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:46.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:46.453+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:46.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:46.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:46.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:46.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:46.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:46.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:46.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:46.440+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:47.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:46.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:47.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:46.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:47.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:46.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:47.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:46.944+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:47.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:47.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:47.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:47.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:47.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:47.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:47.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:47.501+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:47.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:47.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:47.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:47.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:47.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:47.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:47.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:47.446+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:47.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:47.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:47.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:47.935+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:48.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:48.548+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24302 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:48.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:48.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:48.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:48.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:48.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:48.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:48.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:48.459+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:48.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:48.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:48.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:48.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:48.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:48.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:48.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:48.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:49.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:48.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:49.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:48.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:49.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:48.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:49.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:48.920+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:49.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:49.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:49.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:49.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:49.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:49.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:49.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:49.572+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:49.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:49.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:49.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:49.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:49.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:49.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:49.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:49.428+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:50.194 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:49.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:50.195 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:49.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:50.195 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:49.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:50.195 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:49.926+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:50.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:50.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:50.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:50.412+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:50.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:50.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:50.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:50.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:50.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:50.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:50.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:50.538+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:51.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:50.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:51.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:50.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:51.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:50.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:51.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:50.906+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:51.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:51.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:51.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:51.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:51.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:51.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:51.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:51.443+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:51.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:51.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:51.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:51.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:51.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:51.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:51.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:51.584+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:52.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:51.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:52.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:51.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:52.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:51.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:52.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:51.889+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:52.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:52.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:52.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:52.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:52.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:52.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:52.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:52.476+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:52.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:52.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:52.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:52.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:52.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:52.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:52.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:52.597+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:53.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:52.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:53.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:52.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:53.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:52.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:53.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:52.899+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:53.795 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:53.552+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24391 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:53.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:53.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:53.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:53.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:53.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:53.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:53.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:53.435+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:53.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:53.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:53.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:53.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:53.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:53.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:53.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:53.630+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:54.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:53.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:54.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:53.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:54.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:53.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:54.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:53.893+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:54.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:54.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:54.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:54.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:54.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:54.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:54.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:54.613+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:54.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:54.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:54.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:54.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:54.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:54.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:54.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:54.402+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:55.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:54.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:55.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:54.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:55.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:54.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:55.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:54.856+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:55.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:55.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:55.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:55.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:55.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:55.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:55.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:55.382+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:56.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:55.633+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:56.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:55.633+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:56.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:55.633+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:56.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:55.634+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:56.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:55.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:56.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:55.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:56.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:55.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:56.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:55.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:56.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:56.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:56.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:56.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:56.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:56.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:56.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:56.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:56.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:56.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:56.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:56.797 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:56.405+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:57.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:56.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:57.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:56.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:57.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:56.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:57.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:56.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:57.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:57.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:57.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:57.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:57.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:57.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:57.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:57.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:57.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:57.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:57.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:57.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:57.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:57.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:57.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:57.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:58.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:57.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:58.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:57.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:58.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:57.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:58.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:57.801+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:58.686 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:29:58.556+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24481 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:29:58.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:58.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:58.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:58.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:58.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:58.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:58.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:58.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:58.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:58.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:58.688 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:58.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:58.688 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:58.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:58.688 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:58.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:59.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:58.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:59.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:58.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:59.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:58.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:29:59.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:58.777+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:29:59.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:59.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:59.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:59.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:59.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:59.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:29:59.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:29:59.398+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:29:59.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:59.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:59.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:59.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:59.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:59.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:29:59.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:29:59.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:00.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:59.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:00.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:59.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:00.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:59.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:00.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:29:59.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:00.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:00.447+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:00.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:00.447+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:00.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:00.447+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:00.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:00.448+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:00.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:00.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:00.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:00.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:00.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:00.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:00.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:00.597+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:00.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:00.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:00.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:00.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:00.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:00.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:00.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:00.685+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:01.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:01.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:01.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:01.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:01.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:01.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:01.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:01.625+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:01.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:01.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:01.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:01.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:01.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:01.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:01.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:01.430+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:02.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:01.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:02.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:01.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:02.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:01.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:02.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:01.702+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:02.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:02.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:02.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:02.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:02.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:02.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:02.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:02.600+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:02.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:02.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:02.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:02.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:02.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:02.445+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:02.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:02.446+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:03.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:02.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:03.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:02.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:03.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:02.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:03.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:02.724+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:03.689 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:03.560+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24569 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:03.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:03.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:03.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:03.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:03.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:03.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:03.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:03.398+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:03.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:03.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:03.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:03.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:03.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:03.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:03.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:03.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:04.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:03.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:04.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:03.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:04.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:03.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:04.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:03.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:04.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:04.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:04.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:04.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:04.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:04.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:04.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:04.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:05.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:04.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:05.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:04.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:05.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:04.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:05.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:04.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:05.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:04.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:05.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:04.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:05.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:04.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:05.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:04.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:05.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:05.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:05.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:05.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:05.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:05.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:05.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:05.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:05.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:05.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:05.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:05.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:05.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:05.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:05.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:05.606+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:06.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:05.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:06.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:05.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:06.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:05.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:06.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:05.699+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:06.647 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:06.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:06.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:06.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:06.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:06.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:06.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:06.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:07.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:06.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:07.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:06.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:07.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:06.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:07.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:06.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:07.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:06.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:07.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:06.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:07.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:06.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:07.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:06.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:07.662 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:07.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:07.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:07.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:07.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:07.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:07.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:07.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:08.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:07.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:08.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:07.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:08.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:07.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:08.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:07.765+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:08.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:07.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:08.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:07.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:08.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:07.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:08.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:07.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:08.685 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:08.564+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24659 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:08.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:08.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:08.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:08.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:08.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:08.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:08.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:08.348+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:09.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:08.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:09.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:08.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:09.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:08.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:09.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:08.798+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:09.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:08.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:09.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:08.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:09.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:08.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:09.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:08.681+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:09.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:09.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:09.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:09.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:09.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:09.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:09.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:09.307+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:10.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:09.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:10.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:09.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:10.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:09.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:10.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:09.840+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:10.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:09.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:10.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:09.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:10.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:09.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:10.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:09.730+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:10.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:10.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:10.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:10.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:10.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:10.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:10.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:10.289+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:11.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:10.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:11.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:10.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:11.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:10.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:11.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:10.813+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:11.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:10.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:11.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:10.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:11.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:10.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:11.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:10.743+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:11.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:11.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:11.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:11.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:11.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:11.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:11.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:11.285+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:12.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:11.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:12.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:11.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:12.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:11.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:12.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:11.773+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:12.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:11.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:12.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:11.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:12.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:11.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:12.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:11.782+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:12.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:12.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:12.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:12.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:12.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:12.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:12.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:12.305+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:13.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:12.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:13.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:12.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:13.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:12.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:13.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:12.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:13.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:12.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:13.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:12.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:13.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:12.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:13.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:12.756+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:13.583 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:13.568+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24747 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:13.583 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:13.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:13.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:13.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:13.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:13.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:13.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:13.321+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:14.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:13.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:14.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:13.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:14.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:13.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:14.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:13.776+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:14.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:13.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:14.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:13.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:14.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:13.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:14.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:13.717+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:14.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:14.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:14.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:14.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:14.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:14.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:14.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:14.361+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:15.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:14.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:15.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:14.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:15.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:14.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:15.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:14.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:15.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:14.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:15.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:14.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:15.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:14.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:15.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:14.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:15.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:15.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:15.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:15.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:15.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:15.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:15.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:15.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:16.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:15.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:16.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:15.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:16.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:15.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:16.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:15.788+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:16.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:15.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:16.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:15.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:16.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:15.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:16.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:15.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:16.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:16.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:16.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:16.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:16.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:16.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:16.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:16.361+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:17.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:16.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:17.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:16.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:17.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:16.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:17.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:16.806+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:17.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:16.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:17.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:16.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:17.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:16.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:17.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:16.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:17.717 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:17.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:17.717 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:17.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:17.717 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:17.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:17.717 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:17.366+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:18.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:17.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:18.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:17.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:18.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:17.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:18.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:17.847+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:18.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:17.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:18.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:17.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:18.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:17.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:18.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:17.713+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:18.701 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:18.572+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24837 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:18.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:18.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:18.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:18.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:18.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:18.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:18.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:18.398+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:19.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:18.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:19.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:18.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:19.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:18.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:19.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:18.872+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:19.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:18.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:19.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:18.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:19.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:18.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:19.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:18.751+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:19.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:19.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:19.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:19.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:19.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:19.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:19.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:19.435+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:20.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:19.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:20.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:19.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:20.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:19.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:20.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:19.922+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:20.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:19.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:20.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:19.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:20.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:19.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:20.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:19.714+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:20.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:20.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:20.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:20.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:20.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:20.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:20.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:20.440+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:21.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:20.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:21.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:20.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:21.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:20.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:21.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:20.906+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:21.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:20.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:21.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:20.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:21.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:20.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:21.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:20.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:21.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:21.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:21.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:21.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:21.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:21.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:21.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:21.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:22.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:21.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:22.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:21.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:22.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:21.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:22.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:21.898+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:22.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:21.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:22.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:21.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:22.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:21.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:22.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:21.772+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:22.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:22.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:22.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:22.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:22.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:22.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:22.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:22.433+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:23.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:22.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:23.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:22.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:23.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:22.925+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:23.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:22.926+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:23.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:22.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:23.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:22.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:23.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:22.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:23.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:22.757+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:23.704 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:23.576+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 24925 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:23.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:23.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:23.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:23.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:23.705 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:23.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:23.705 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:23.423+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:24.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:23.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:24.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:23.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:24.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:23.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:24.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:23.934+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:24.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:23.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:24.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:23.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:24.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:23.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:24.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:23.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:24.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:24.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:24.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:24.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:24.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:24.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:24.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:24.421+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:25.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:24.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:25.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:24.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:25.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:24.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:25.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:24.684+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:25.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:24.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:25.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:24.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:25.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:24.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:25.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:24.909+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:25.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:25.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:25.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:25.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:25.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:25.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:25.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:25.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:25.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:25.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:25.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:25.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:25.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:25.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:25.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:25.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:26.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:25.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:26.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:25.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:26.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:25.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:26.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:25.953+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:26.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:26.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:26.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:26.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:26.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:26.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:26.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:26.344+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:27.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:26.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:27.006 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:26.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:27.006 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:26.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:27.006 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:26.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:27.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:27.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:27.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:27.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:27.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:27.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:27.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:27.002+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:27.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:27.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:27.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:27.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:27.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:27.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:27.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:27.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:28.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:27.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:28.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:27.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:28.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:27.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:28.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:27.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:28.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:28.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:28.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:28.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:28.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:28.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:28.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:28.014+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:28.641 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:28.580+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25015 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:28.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:28.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:28.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:28.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:28.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:28.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:28.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:28.383+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:29.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:28.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:29.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:28.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:29.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:28.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:29.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:28.638+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:29.014+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:29.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:29.014+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:29.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:29.014+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:29.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:29.015+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:29.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:29.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:29.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:29.632+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:29.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:29.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:29.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:29.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:29.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:29.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:29.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:29.388+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:30.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:29.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:30.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:29.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:30.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:29.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:30.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:30.000+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:30.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:30.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:30.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:30.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:30.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:30.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:30.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:30.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:30.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:30.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:30.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:30.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:30.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:30.428+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:31.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:31.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:31.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:31.009+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:31.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:31.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:31.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:31.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:31.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:31.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:31.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:31.579+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:31.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:31.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:31.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:31.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:31.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:31.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:31.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:31.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:32.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:32.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:32.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:32.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:32.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:32.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:32.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:32.037+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:32.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:32.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:32.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:32.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:32.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:32.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:32.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:32.371+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:32.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:33.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:32.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:33.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:32.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:33.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:32.599+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:33.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:33.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:33.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:33.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:33.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:33.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:33.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:33.071+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:33.712 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:33.584+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25104 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:33.712 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:33.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:33.712 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:33.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:33.713 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:33.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:33.713 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:33.402+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:33.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:33.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:33.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:33.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:33.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:33.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:33.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:33.614+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:34.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:34.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:34.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:34.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:34.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:34.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:34.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:34.118+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:34.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:34.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:34.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:34.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:34.415+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:34.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:34.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:34.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:34.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:34.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:34.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:34.610+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:35.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:35.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:35.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:35.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:35.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:35.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:35.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:35.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:35.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:35.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:35.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:35.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:35.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:35.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:35.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:35.154+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:36.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:35.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:36.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:35.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:36.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:35.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:36.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:35.598+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:36.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:36.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:36.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:36.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:36.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:36.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:36.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:36.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:36.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:36.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:36.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:36.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:36.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:36.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:36.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:36.188+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:37.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:36.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:37.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:36.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:37.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:36.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:37.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:36.617+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:37.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:37.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:37.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:37.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:37.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:37.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:37.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:37.441+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:37.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:37.213+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:37.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:37.213+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:37.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:37.213+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:37.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:37.214+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:37.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:38.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:37.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:38.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:37.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:38.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:37.657+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:38.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:38.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:38.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:38.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:38.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:38.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:38.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:38.210+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:38.794 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:38.588+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25194 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:38.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:38.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:38.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:38.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:38.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:38.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:38.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:38.471+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:38.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:38.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:38.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:38.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:38.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:38.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:38.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:38.636+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:39.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:39.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:39.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:39.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:39.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:39.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:39.176+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:39.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:39.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:39.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:39.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:39.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:39.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:39.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:39.513+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:39.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:39.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:39.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:39.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:39.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:39.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:40.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:40.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:40.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:40.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:40.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:40.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:40.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:40.160+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:40.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:40.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:40.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:40.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:40.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:40.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:40.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:40.714+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:40.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:40.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:40.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:40.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:40.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:40.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:40.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:40.507+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:41.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:41.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:41.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:41.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:41.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:41.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:41.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:41.191+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:41.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:41.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:41.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:41.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:41.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:41.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:41.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:41.527+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:41.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:41.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:41.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:41.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:41.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:41.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:41.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:41.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:42.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:42.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:42.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:42.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:42.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:42.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:42.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:42.170+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:42.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:42.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:42.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:42.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:42.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:42.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:42.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:42.478+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:42.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:42.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:42.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:42.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:42.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:42.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:43.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:43.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:43.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:43.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:43.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:43.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:43.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:43.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:43.794 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:43.592+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25281 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:43.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:43.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:43.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:43.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:43.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:43.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:43.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:43.663+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:43.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:43.496+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:43.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:43.496+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:43.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:43.496+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:43.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:43.497+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:44.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:44.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:44.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:44.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:44.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:44.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:44.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:44.132+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:44.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:44.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:44.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:44.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:44.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:44.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:44.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:44.461+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:44.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:44.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:44.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:44.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:44.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:44.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:44.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:44.682+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:45.487 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:45.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:45.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:45.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:45.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:45.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:45.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:45.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:45.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:45.484+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:45.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:45.484+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:45.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:45.484+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:45.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:45.484+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:45.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:45.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:45.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:45.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:45.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:46.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:46.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:46.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:46.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:46.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:46.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:46.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:46.128+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:46.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:46.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:46.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:46.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:46.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:46.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:46.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:46.465+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:46.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:46.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:46.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:46.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:46.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:46.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:46.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:46.665+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:47.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:47.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:47.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:47.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:47.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:47.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:47.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:47.100+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:47.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:47.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:47.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:47.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:47.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:47.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:47.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:47.701+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:47.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:47.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:47.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:47.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:47.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:47.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:47.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:47.513+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:48.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:48.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:48.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:48.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:48.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:48.099+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:49.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:48.596+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25370 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:48.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:48.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:48.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:48.697+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:49.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:48.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:49.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:48.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:49.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:48.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:49.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:48.546+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:49.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:49.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:49.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:49.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:49.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:49.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:49.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:49.142+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:49.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:49.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:49.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:49.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:49.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:49.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:49.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:49.500+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:49.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:49.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:49.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:49.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:49.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:49.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:49.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:49.723+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:50.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:50.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:50.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:50.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:50.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:50.169+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:50.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:50.169+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:51.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:50.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:51.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:50.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:51.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:50.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:51.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:50.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:50.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:50.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:51.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:50.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:51.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:50.728+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:51.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:51.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:51.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:51.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:51.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:51.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:51.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:51.170+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:52.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:51.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:52.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:51.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:52.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:51.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:52.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:51.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:52.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:51.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:52.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:51.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:52.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:51.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:52.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:51.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:52.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:52.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:52.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:52.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:52.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:52.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:52.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:52.150+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:53.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:52.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:53.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:52.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:53.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:52.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:53.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:52.606+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:53.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:52.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:53.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:52.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:53.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:52.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:53.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:52.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:53.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:53.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:53.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:53.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:53.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:53.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:53.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:53.170+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:54.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:53.600+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25459 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:54.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:53.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:54.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:53.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:54.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:53.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:54.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:53.590+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:53.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:53.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:54.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:53.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:54.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:53.741+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:54.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:54.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:54.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:54.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:54.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:54.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:54.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:54.188+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:55.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:54.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:55.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:54.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:55.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:54.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:55.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:54.545+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:55.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:54.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:55.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:54.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:55.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:54.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:55.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:54.773+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:55.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:55.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:55.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:55.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:55.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:55.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:55.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:55.166+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:55.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:55.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:55.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:55.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:55.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:55.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:55.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:55.527+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:55.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:55.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:55.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:55.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:55.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:55.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:55.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:55.741+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:56.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:56.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:56.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:56.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:56.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:56.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:56.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:56.128+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:56.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:56.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:56.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:56.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:56.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:56.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:56.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:56.500+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:57.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:56.775+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:57.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:56.775+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:57.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:56.775+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:57.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:56.775+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:57.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:57.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:57.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:57.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:57.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:57.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:57.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:57.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:57.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:57.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:57.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:57.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:57.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:57.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:57.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:57.461+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:58.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:57.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:58.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:57.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:58.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:57.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:58.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:57.728+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:58.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:58.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:58.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:58.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:58.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:58.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:58.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:58.082+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:58.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:30:58.605+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25549 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:30:58.736 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:58.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:58.736 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:58.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:58.736 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:58.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:58.736 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:58.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:30:59.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:58.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:59.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:58.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:59.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:58.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:30:59.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:58.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:59.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:59.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:59.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:59.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:59.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:59.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:30:59.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:30:59.084+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:30:59.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:59.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:59.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:59.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:59.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:59.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:30:59.719 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:30:59.450+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:00.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:00.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:00.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:00.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:30:59.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:00.477 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:00.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:00.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:00.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:00.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:00.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:00.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:00.103+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:00.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:00.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:00.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:00.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:00.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:00.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:00.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:00.474+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:01.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:00.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:01.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:00.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:01.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:00.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:01.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:00.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:01.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:01.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:01.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:01.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:01.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:01.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:01.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:01.122+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:01.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:01.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:01.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:01.523+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:02.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:01.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:02.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:01.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:02.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:01.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:02.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:01.782+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:02.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:02.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:02.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:02.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:02.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:02.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:02.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:02.157+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:02.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:02.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:02.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:02.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:02.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:02.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:02.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:02.507+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:03.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:02.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:03.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:02.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:03.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:02.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:03.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:02.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:03.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:03.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:03.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:03.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:03.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:03.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:03.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:03.176+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:03.794 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:03.608+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25637 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:03.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:03.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:03.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:03.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:03.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:03.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:03.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:03.742+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:03.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:03.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:03.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:03.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:03.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:03.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:03.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:03.515+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:04.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:04.189+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:04.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:04.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:04.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:04.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:04.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:04.190+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:04.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:04.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:04.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:04.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:04.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:04.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:04.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:04.519+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:04.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:04.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:04.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:04.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:04.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:04.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:04.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:04.753+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:05.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:05.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:05.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:05.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:05.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:05.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:05.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:05.175+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:05.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:05.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:05.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:05.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:05.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:05.508+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:05.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:05.509+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:05.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:05.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:05.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:05.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:05.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:05.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:05.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:05.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:06.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:06.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:06.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:06.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:06.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:06.200+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:07.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:06.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:07.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:06.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:07.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:06.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:07.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:06.668+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:07.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:06.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:07.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:06.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:07.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:06.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:07.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:06.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:07.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:07.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:07.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:07.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:07.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:07.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:07.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:07.176+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:08.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:07.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:08.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:07.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:08.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:07.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:08.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:07.648+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:08.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:07.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:08.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:07.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:08.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:07.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:08.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:07.565+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:08.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:08.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:08.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:08.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:08.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:08.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:08.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:08.216+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:08.604 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:08.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:08.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:08.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:08.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:08.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:08.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:08.545+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:09.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:08.612+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25727 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:09.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:08.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:09.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:08.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:09.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:08.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:09.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:08.601+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:09.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:09.171+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:09.171+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:09.171+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:09.171+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:09.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:09.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:09.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:09.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:09.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:09.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:09.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:09.502+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:09.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:09.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:09.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:09.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:09.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:09.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:09.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:09.649+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:10.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:10.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:10.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:10.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:10.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:10.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:10.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:10.128+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:10.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:10.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:10.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:10.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:10.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:10.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:10.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:10.462+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:10.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:10.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:10.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:10.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:10.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:10.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:10.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:10.647+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:11.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:11.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:11.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:11.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:11.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:11.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:11.447 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:11.156+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:11.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:11.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:11.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:11.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:11.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:11.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:11.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:11.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:11.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:11.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:11.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:11.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:11.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:11.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:11.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:11.443+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:12.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:12.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:12.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:12.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:12.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:12.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:12.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:12.143+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:12.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:12.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:12.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:12.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:12.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:12.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:12.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:12.467+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:12.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:12.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:12.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:12.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:12.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:12.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:12.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:12.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:13.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:13.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:13.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:13.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:13.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:13.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:13.433 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:13.148+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:13.715 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:13.616+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25815 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:13.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:13.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:13.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:13.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:13.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:13.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:13.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:13.429+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:14.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:13.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:14.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:13.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:14.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:13.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:14.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:13.712+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:14.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:14.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:14.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:14.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:14.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:14.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:14.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:14.139+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:14.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:14.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:14.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:14.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:14.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:14.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:14.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:14.466+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:15.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:14.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:15.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:14.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:15.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:14.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:15.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:14.745+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:15.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:15.182+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:15.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:15.182+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:15.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:15.182+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:15.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:15.183+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:15.786 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:15.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:15.786 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:15.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:15.786 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:15.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:15.786 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:15.503+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:16.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:15.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:16.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:15.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:16.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:15.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:16.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:15.783+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:16.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:16.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:16.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:16.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:16.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:16.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:16.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:16.145+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:16.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:16.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:16.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:16.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:16.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:16.509+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:16.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:16.510+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:17.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:16.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:17.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:16.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:17.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:16.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:17.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:16.763+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:17.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:17.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:17.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:17.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:17.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:17.125+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:18.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:17.560+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:18.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:17.560+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:18.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:17.560+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:18.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:17.561+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:18.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:17.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:18.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:17.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:18.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:17.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:18.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:17.807+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:18.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:18.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:18.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:18.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:18.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:18.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:18.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:18.103+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:19.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:18.620+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25904 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:19.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:18.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:19.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:18.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:19.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:18.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:19.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:18.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:19.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:18.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:19.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:18.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:19.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:18.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:19.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:18.584+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:19.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:19.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:19.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:19.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:19.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:19.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:19.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:19.149+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:19.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:19.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:19.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:19.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:19.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:19.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:19.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:19.624+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:20.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:19.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:20.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:19.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:20.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:19.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:20.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:19.786+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:20.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:20.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:20.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:20.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:20.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:20.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:21.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:20.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:21.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:20.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:21.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:20.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:21.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:20.674+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:21.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:20.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:21.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:20.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:21.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:20.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:21.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:20.799+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:21.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:21.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:21.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:21.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:21.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:21.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:21.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:21.083+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:22.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:21.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:22.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:21.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:22.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:21.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:22.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:21.658+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:22.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:21.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:22.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:21.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:22.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:21.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:22.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:21.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:22.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:22.097+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:22.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:22.097+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:22.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:22.097+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:22.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:22.097+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:23.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:22.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:23.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:22.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:23.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:22.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:23.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:22.621+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:23.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:22.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:23.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:22.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:23.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:22.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:23.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:22.788+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:23.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:23.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:23.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:23.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:23.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:23.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:23.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:23.083+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:23.987 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:23.625+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 25992 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:23.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:23.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:23.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:23.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:23.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:23.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:23.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:23.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:23.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:23.652+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:23.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:23.652+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:23.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:23.652+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:23.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:23.653+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:24.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:24.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:24.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:24.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:24.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:24.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:24.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:24.133+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:25.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:24.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:25.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:24.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:25.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:24.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:25.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:24.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:25.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:24.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:25.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:24.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:25.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:24.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:25.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:24.626+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:25.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:25.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:25.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:25.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:25.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:25.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:25.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:25.092+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:26.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:25.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:26.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:25.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:26.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:25.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:26.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:25.772+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:25.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:25.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:25.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:25.587+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:26.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:26.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:26.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:26.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:26.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:26.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:26.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:26.118+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:27.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:26.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:27.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:26.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:27.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:26.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:27.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:26.734+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:26.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:26.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:26.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:26.567+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:27.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:27.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:27.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:27.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:27.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:27.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:27.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:27.113+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:28.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:27.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:28.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:27.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:28.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:27.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:28.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:27.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:28.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:27.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:28.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:27.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:28.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:27.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:28.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:27.756+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:28.067+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:28.067+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:28.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:28.067+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:28.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:28.068+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:29.034 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:28.629+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26081 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:29.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:28.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:29.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:28.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:29.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:28.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:29.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:28.554+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:29.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:28.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:29.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:28.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:29.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:28.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:29.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:28.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:29.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:29.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:29.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:29.032+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:29.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:29.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:29.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:29.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:29.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:29.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:29.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:29.578+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:29.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:29.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:29.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:29.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:29.741+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:30.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:29.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:30.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:29.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:30.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:29.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:30.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:29.986+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:30.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:30.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:30.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:30.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:30.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:30.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:30.541+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:31.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:30.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:30.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:30.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:31.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:30.954+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:31.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:30.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:31.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:30.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:31.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:30.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:31.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:30.792+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:31.973 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:31.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:31.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:31.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:31.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:31.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:31.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:31.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:31.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:31.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:31.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:31.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:31.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:31.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:31.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:31.589+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:32.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:31.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:32.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:31.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:32.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:31.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:32.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:31.971+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:32.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:32.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:32.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:32.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:32.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:32.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:32.943 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:32.604+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:32.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:32.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:32.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:32.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:32.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:32.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:32.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:32.836+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:33.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:32.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:33.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:32.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:33.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:32.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:33.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:32.939+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:33.903 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:33.633+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26170 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:33.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:33.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:33.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:33.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:33.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:33.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:33.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:33.617+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:33.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:33.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:33.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:33.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:33.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:33.807+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:33.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:33.807+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:34.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:33.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:34.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:33.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:34.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:33.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:34.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:33.900+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:34.886 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:34.622+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:34.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:34.622+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:34.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:34.622+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:34.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:34.623+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:34.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:34.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:34.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:34.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:34.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:34.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:34.888 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:34.788+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:35.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:34.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:34.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:34.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:35.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:34.884+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:35.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:35.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:35.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:35.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:35.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:35.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:35.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:35.640+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:35.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:35.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:35.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:35.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:35.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:35.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:35.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:35.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:36.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:35.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:36.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:35.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:36.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:35.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:36.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:35.928+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:36.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:36.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:36.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:36.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:36.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:36.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:36.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:36.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:36.967 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:36.967 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:36.968 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:36.968 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:36.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:37.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:36.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:37.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:36.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:37.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:36.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:37.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:36.964+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:37.956 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:37.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:37.956 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:37.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:37.956 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:37.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:37.956 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:37.676+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:37.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:37.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:37.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:37.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:37.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:37.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:37.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:37.747+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:38.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:37.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:38.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:37.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:38.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:37.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:38.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:37.953+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:38.919 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:38.639+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26260 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:38.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:38.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:38.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:38.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:38.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:38.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:38.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:38.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:38.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:38.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:38.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:38.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:38.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:38.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:38.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:38.726+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:39.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:38.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:39.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:38.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:39.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:38.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:39.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:38.918+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:39.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:39.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:39.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:39.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:39.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:39.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:39.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:39.657+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:39.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:39.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:39.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:39.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:39.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:40.201 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:39.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:40.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:39.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:40.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:39.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:40.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:39.868+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:41.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:40.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:41.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:40.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:41.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:40.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:41.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:40.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:41.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:41.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:41.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:41.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:40.734+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:41.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:40.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:41.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:40.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:41.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:40.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:41.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:40.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:41.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:41.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:42.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:41.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:42.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:41.640+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:42.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:41.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:42.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:41.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:42.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:41.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:42.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:41.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:42.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:41.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:42.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:41.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:42.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:41.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:42.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:41.857+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:43.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:42.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:43.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:42.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:43.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:42.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:43.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:42.701+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:43.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:42.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:43.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:42.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:43.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:42.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:43.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:42.673+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:43.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:42.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:43.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:42.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:43.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:42.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:43.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:42.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:43.877 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:43.642+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26348 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:43.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:43.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:43.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:43.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:43.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:43.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:43.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:43.736+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:43.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:43.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:43.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:43.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:43.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:43.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:43.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:43.722+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:43.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:43.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:43.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:43.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:43.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:43.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:43.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:43.873+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:45.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:44.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:45.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:44.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:45.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:44.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:45.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:44.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:45.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:44.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:45.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:44.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:45.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:44.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:45.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:44.728+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:45.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:44.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:45.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:44.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:45.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:44.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:45.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:44.862+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:45.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:45.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:45.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:45.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:45.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:45.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:45.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:45.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:45.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:45.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:45.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:45.736+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:46.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:45.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:46.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:45.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:46.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:45.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:46.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:45.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:47.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:46.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:47.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:46.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:47.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:46.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:47.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:46.790+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:46.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:46.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:46.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:46.698+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:47.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:46.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:47.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:46.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:47.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:46.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:47.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:46.691+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:48.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:47.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:48.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:47.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:48.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:47.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:48.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:47.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:48.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:47.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:48.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:47.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:48.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:47.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:48.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:47.711+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:48.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:47.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:48.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:47.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:48.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:47.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:48.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:47.733+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:49.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:48.646+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26438 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:49.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:48.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:49.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:48.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:49.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:48.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:49.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:48.857+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:48.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:49.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:48.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:49.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:48.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:49.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:48.741+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:49.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:48.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:49.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:48.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:49.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:48.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:49.047 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:48.695+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:49.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:49.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:49.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:49.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:49.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:49.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:49.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:49.672+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:49.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:49.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:49.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:49.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:49.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:49.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:49.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:49.742+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:50.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:49.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:50.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:49.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:50.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:49.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:50.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:49.860+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:51.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:50.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:51.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:50.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:51.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:50.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:51.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:50.875+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:50.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:50.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:50.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:51.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:50.773+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:51.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:50.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:51.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:50.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:51.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:50.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:51.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:50.720+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:52.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:51.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:52.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:51.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:52.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:51.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:52.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:51.869+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:52.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:51.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:52.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:51.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:52.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:51.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:52.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:51.770+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:52.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:51.677+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:52.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:51.677+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:52.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:51.677+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:52.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:51.678+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:52.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:52.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:52.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:52.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:52.915 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:52.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:52.915 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:52.649+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:52.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:52.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:52.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:52.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:52.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:52.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:52.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:52.764+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:53.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:52.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:53.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:52.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:53.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:52.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:53.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:52.912+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:53.927 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:53.650+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26526 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:53.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:53.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:53.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:53.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:53.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:53.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:53.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:53.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:53.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:53.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:53.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:53.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:53.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:53.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:53.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:53.647+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:54.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:53.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:54.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:53.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:54.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:53.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:54.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:53.925+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:54.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:54.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:54.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:54.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:54.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:54.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:54.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:54.749+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:54.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:54.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:54.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:54.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:54.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:54.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:54.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:54.605+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:55.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:54.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:55.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:54.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:55.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:54.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:55.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:54.928+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:55.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:55.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:55.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:55.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:55.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:55.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:55.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:55.734+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:55.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:55.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:55.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:55.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:55.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:55.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:55.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:55.647+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:55.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:55.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:55.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:55.931+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:57.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:56.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:57.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:56.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:57.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:56.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:57.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:56.907+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:57.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:56.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:57.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:56.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:57.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:56.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:57.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:56.674+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:57.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:56.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:57.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:56.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:57.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:56.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:57.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:56.701+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:57.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:57.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:57.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:57.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:57.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:57.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:57.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:57.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:57.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:57.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:57.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:57.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:57.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:57.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:57.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:57.717+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:57.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:57.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:57.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:57.948+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:58.963 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:31:58.655+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26616 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:31:58.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:58.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:58.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:58.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:58.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:58.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:58.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:58.702+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:31:58.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:58.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:58.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:58.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:58.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:58.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:58.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:58.692+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:58.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:58.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:58.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:31:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:58.961+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:59.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:59.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:59.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:31:59.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:31:59.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:31:59.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:59.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:59.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:59.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:59.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:59.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:31:59.796 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:31:59.700+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:00.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:59.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:00.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:59.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:00.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:59.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:00.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:31:59.978+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:00.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:00.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:00.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:00.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:00.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:00.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:00.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:00.661+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:00.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:00.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:00.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:00.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:00.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:00.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:00.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:00.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:01.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:00.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:01.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:00.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:01.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:00.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:01.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:00.979+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:01.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:01.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:01.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:01.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:01.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:01.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:01.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:01.706+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:01.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:01.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:01.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:01.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:01.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:01.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:01.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:01.690+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:02.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:01.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:02.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:01.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:02.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:01.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:02.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:01.958+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:02.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:02.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:02.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:02.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:02.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:02.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:02.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:02.721+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:02.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:02.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:02.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:02.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:02.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:02.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:02.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:02.681+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:03.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:02.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:03.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:02.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:03.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:02.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:03.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:02.962+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:03.923 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:03.660+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26703 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:03.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:03.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:03.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:03.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:03.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:03.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:03.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:03.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:03.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:03.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:03.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:03.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:03.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:03.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:03.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:03.657+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:04.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:03.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:04.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:03.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:04.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:03.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:04.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:03.978+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:04.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:04.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:04.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:04.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:04.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:04.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:04.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:04.631+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:04.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:04.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:04.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:04.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:04.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:04.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:04.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:04.706+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:05.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:04.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:05.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:04.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:05.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:04.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:05.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:04.942+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:05.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:05.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:05.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:05.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:05.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:05.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:05.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:05.676+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:05.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:05.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:05.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:05.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:05.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:05.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:05.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:05.591+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:06.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:05.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:06.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:05.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:06.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:05.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:06.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:05.909+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:06.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:06.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:06.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:06.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:06.921 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:06.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:06.921 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:06.648+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:06.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:06.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:06.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:06.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:06.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:06.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:06.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:06.603+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:07.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:06.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:07.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:06.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:07.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:06.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:07.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:06.917+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:07.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:07.645+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:07.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:07.645+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:07.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:07.645+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:07.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:07.645+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:07.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:07.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:07.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:07.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:07.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:07.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:07.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:07.693+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:08.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:07.899+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:08.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:07.899+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:08.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:07.899+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:08.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:07.900+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:08.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:08.664+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26792 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:08.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:08.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:08.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:08.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:08.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:08.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:08.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:08.699+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:08.886 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:08.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:08.886 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:08.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:08.886 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:08.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:08.886 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:08.622+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:09.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:08.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:09.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:08.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:09.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:08.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:09.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:08.882+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:09.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:09.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:09.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:09.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:09.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:09.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:09.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:09.684+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:09.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:09.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:09.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:09.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:09.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:09.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:09.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:09.627+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:10.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:09.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:10.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:09.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:10.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:09.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:10.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:09.863+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:11.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:10.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:11.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:10.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:11.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:10.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:11.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:10.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:11.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:10.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:11.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:10.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:11.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:10.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:11.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:10.885+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:11.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:10.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:11.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:10.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:11.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:10.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:11.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:10.678+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:12.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:11.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:12.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:11.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:12.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:11.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:12.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:11.889+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:12.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:11.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:12.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:11.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:12.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:11.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:12.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:11.659+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:12.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:11.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:12.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:11.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:12.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:11.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:12.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:11.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:13.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:12.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:13.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:12.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:13.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:12.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:13.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:12.701+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:13.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:12.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:13.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:12.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:13.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:12.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:13.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:12.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:13.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:12.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:13.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:12.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:12.924+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:13.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:12.924+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:13.949 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:13.668+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26881 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:13.949 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:13.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:13.949 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:13.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:13.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:13.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:13.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:13.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:13.950 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:13.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:13.950 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:13.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:13.950 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:13.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:13.950 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:13.697+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:13.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:13.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:13.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:13.946+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:15.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:14.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:15.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:14.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:15.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:14.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:15.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:14.735+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:15.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:14.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:15.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:14.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:15.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:14.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:15.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:14.957+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:15.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:14.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:15.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:14.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:15.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:14.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:15.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:14.735+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:16.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:15.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:16.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:15.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:16.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:15.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:16.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:15.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:16.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:15.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:16.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:15.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:16.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:15.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:16.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:15.923+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:16.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:15.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:16.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:15.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:16.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:15.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:16.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:15.726+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:17.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:16.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:17.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:16.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:17.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:16.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:17.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:16.753+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:17.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:16.935+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:17.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:16.935+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:17.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:16.935+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:17.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:16.935+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:17.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:16.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:17.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:16.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:17.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:16.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:17.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:16.728+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:17.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:17.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:17.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:17.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:17.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:17.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:17.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:17.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:17.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:17.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:17.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:17.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:17.963 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:17.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:17.963 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:17.687+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:18.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:17.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:18.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:17.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:18.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:17.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:18.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:17.958+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:18.981 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:18.673+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 26971 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:18.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:18.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:18.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:18.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:18.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:18.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:18.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:18.719+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:18.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:18.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:18.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:18.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:18.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:18.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:18.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:18.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:18.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:18.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:18.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:18.978+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:19.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:19.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:19.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:19.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:19.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:19.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:19.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:19.686+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:20.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:19.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:20.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:19.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:20.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:19.969+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:20.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:19.969+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:20.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:19.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:20.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:19.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:20.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:19.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:20.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:19.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:20.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:20.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:20.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:20.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:20.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:20.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:20.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:20.758+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:20.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:20.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:20.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:20.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:20.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:20.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:20.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:20.680+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:21.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:20.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:21.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:20.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:21.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:20.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:21.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:20.932+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:21.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:21.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:21.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:21.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:21.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:21.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:21.943 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:21.669+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:21.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:21.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:21.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:21.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:21.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:21.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:21.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:21.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:22.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:21.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:22.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:21.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:22.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:21.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:22.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:21.940+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:23.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:22.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:23.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:22.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:23.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:22.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:23.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:22.690+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:23.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:22.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:23.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:22.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:23.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:22.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:23.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:22.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:23.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:22.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:23.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:22.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:23.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:22.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:23.046 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:22.903+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:24.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:23.677+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27059 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:24.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:23.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:24.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:23.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:24.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:23.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:24.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:23.861+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:24.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:23.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:24.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:23.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:24.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:23.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:24.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:23.658+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:24.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:23.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:24.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:23.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:24.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:23.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:24.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:23.765+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:24.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:24.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:24.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:24.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:24.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:24.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:24.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:24.629+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:24.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:24.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:24.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:24.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:24.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:24.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:24.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:24.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:25.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:24.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:25.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:24.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:25.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:24.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:25.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:24.896+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:26.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:25.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:26.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:25.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:26.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:25.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:26.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:25.878+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:25.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:25.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:25.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:25.679+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:26.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:25.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:26.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:25.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:26.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:25.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:26.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:25.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:27.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:26.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:27.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:26.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:27.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:26.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:27.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:26.927+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:26.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:26.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:26.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:26.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:27.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:26.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:27.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:26.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:27.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:26.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:27.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:26.804+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:27.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:27.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:27.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:27.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:27.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:27.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:27.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:27.701+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:27.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:27.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:27.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:27.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:27.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:28.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:27.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:28.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:27.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:28.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:27.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:28.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:27.976+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:29.016 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:28.682+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27149 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:29.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:28.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:29.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:28.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:29.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:28.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:29.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:28.737+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:29.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:28.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:29.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:28.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:29.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:28.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:29.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:28.797+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:29.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:29.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:29.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:29.014+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:29.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:29.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:29.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:29.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:29.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:29.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:29.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:29.721+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:30.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:29.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:30.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:29.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:30.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:29.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:30.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:29.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:30.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:30.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:30.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:30.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:30.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:30.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:30.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:30.045+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:31.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:30.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:31.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:30.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:31.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:30.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:31.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:30.702+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:31.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:30.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:31.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:30.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:31.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:30.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:31.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:30.732+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:31.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:31.049+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:31.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:31.049+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:31.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:31.049+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:31.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:31.050+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:32.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:31.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:32.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:31.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:32.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:31.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:32.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:31.743+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:32.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:31.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:32.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:31.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:32.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:31.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:32.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:31.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:32.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:32.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:32.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:32.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:32.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:32.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:32.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:33.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:32.774+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:33.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:32.734+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:33.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:32.734+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:33.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:32.734+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:33.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:32.735+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:33.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:33.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:33.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:33.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:33.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:33.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:33.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:33.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:34.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:33.686+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27238 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:34.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:33.742+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:34.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:33.742+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:34.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:33.742+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:34.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:33.743+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:34.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:33.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:34.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:33.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:34.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:33.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:34.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:33.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:34.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:34.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:34.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:34.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:34.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:34.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:34.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:34.104+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:35.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:34.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:35.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:34.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:35.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:34.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:35.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:34.760+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:35.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:34.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:35.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:34.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:35.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:34.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:35.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:34.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:35.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:35.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:35.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:35.134+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:36.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:35.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:36.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:35.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:36.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:35.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:36.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:35.776+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:36.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:35.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:36.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:35.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:36.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:35.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:36.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:35.729+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:36.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:36.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:36.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:36.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:36.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:36.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:36.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:36.102+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:37.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:36.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:37.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:36.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:37.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:36.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:37.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:36.757+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:37.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:36.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:37.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:36.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:37.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:36.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:37.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:36.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:37.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:37.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:37.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:37.100+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:38.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:37.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:38.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:37.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:38.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:37.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:38.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:37.797+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:38.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:37.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:38.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:37.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:38.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:37.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:38.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:37.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:38.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:38.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:38.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:38.099+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:39.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:38.691+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27327 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:39.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:38.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:39.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:38.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:39.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:38.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:39.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:38.819+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:39.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:38.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:39.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:38.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:39.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:38.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:39.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:38.772+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:39.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:39.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:39.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:39.094+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:39.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:39.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:39.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:39.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:39.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:39.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:39.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:39.736+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:40.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:39.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:40.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:39.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:40.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:39.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:40.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:39.787+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:40.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:40.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:40.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:40.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:40.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:40.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:40.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:40.072+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:41.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:40.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:41.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:40.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:41.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:40.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:41.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:40.761+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:41.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:40.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:41.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:40.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:41.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:40.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:41.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:40.792+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:41.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:41.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:41.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:41.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:41.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:41.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:41.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:41.027+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:41.780+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:41.780+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:41.780+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:41.781+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:42.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:41.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:42.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:41.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:42.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:41.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:42.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:41.751+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:42.055+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:42.055+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:42.055+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:42.056+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:43.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:42.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:43.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:42.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:43.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:42.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:43.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:42.829+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:43.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:42.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:43.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:42.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:43.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:42.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:43.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:42.735+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:43.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:43.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:43.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:43.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:43.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:43.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:43.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:43.061+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:44.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:43.696+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27415 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:44.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:43.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:44.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:43.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:44.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:43.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:44.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:43.804+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:44.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:43.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:44.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:43.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:44.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:43.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:44.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:43.777+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:44.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:44.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:44.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:44.088+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:45.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:44.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:45.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:44.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:45.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:44.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:45.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:44.826+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:45.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:44.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:45.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:44.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:45.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:44.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:45.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:44.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:45.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:45.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:45.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:45.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:45.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:45.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:45.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:45.062+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:45.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:45.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:45.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:46.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:45.805+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:46.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:45.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:46.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:45.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:46.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:45.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:46.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:45.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:46.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:46.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:46.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:46.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:46.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:46.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:46.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:46.061+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:46.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:46.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:46.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:46.763+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:46.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:46.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:46.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:47.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:46.803+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:47.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:47.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:47.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:47.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:47.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:47.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:47.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:47.041+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:48.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:47.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:48.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:47.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:48.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:47.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:48.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:47.805+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:48.074 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:47.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:48.074 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:47.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:48.074 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:47.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:48.074 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:47.841+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:48.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:48.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:48.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:48.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:49.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:48.700+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27505 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:49.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:48.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:49.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:48.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:49.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:48.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:49.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:48.815+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:48.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:48.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:48.769+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:49.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:48.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:49.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:49.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:49.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:49.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:49.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:49.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:49.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:49.094+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:50.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:49.837+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:50.104 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:49.837+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:50.104 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:49.837+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:50.104 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:49.838+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:50.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:49.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:50.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:49.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:50.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:49.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:50.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:49.804+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:50.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:50.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:50.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:50.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:50.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:50.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:50.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:50.101+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:51.078 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:50.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:51.079 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:50.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:51.079 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:50.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:51.079 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:50.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:51.079 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:50.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:51.079 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:50.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:51.080 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:50.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:51.080 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:50.799+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:51.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:51.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:51.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:51.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:51.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:51.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:51.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:51.076+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:52.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:51.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:52.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:51.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:52.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:51.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:52.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:51.858+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:52.129 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:51.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:52.129 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:51.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:52.129 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:51.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:52.129 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:51.833+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:52.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:52.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:52.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:52.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:52.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:52.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:52.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:52.125+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:53.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:52.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:53.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:52.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:53.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:52.825+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:53.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:52.825+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:53.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:52.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:53.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:52.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:53.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:52.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:53.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:52.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:53.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:53.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:53.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:53.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:53.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:53.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:53.087+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:54.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:53.705+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27594 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:54.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:53.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:54.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:53.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:54.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:53.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:54.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:53.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:53.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:53.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:53.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:53.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:54.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:54.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:54.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:54.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:54.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:54.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:54.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:54.123+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:55.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:54.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:55.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:54.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:55.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:54.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:55.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:54.804+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:55.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:54.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:55.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:54.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:55.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:54.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:55.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:54.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:55.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:55.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:55.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:55.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:55.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:55.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:55.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:55.113+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:56.163 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:55.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:56.163 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:55.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:56.163 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:55.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:56.163 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:55.848+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:56.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:55.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:56.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:55.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:56.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:55.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:56.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:55.878+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:56.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:56.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:56.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:56.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:56.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:56.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:56.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:56.160+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:57.182 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:56.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:57.182 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:56.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:57.182 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:56.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:57.183 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:56.925+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:57.183 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:56.815+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:57.183 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:56.815+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:57.183 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:56.815+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:57.183 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:56.815+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:57.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:57.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:57.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:57.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:57.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:57.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:57.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:57.180+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:58.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:57.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:58.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:57.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:58.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:57.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:58.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:57.915+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:58.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:57.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:58.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:57.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:58.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:57.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:58.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:57.791+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:58.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:58.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:58.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:58.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:58.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:58.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:58.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:58.146+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:59.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:32:58.709+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27683 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:32:59.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:58.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:59.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:58.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:59.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:58.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:32:59.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:58.944+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:32:59.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:58.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:59.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:58.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:59.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:58.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:32:59.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:58.823+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:32:59.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:59.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:59.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:59.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:59.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:59.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:32:59.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:32:59.162+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:00.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:59.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:00.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:59.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:00.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:59.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:00.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:32:59.901+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:00.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:59.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:00.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:59.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:00.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:59.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:00.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:32:59.785+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:00.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:00.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:00.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:00.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:00.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:00.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:00.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:00.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:01.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:00.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:01.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:00.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:01.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:00.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:01.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:00.902+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:01.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:00.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:01.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:00.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:01.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:00.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:01.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:00.738+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:01.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:01.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:01.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:01.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:01.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:01.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:01.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:01.142+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:02.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:01.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:02.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:01.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:02.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:01.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:02.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:01.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:02.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:01.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:02.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:01.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:02.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:01.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:02.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:01.778+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:02.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:02.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:02.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:02.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:03.177 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:02.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:03.177 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:02.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:03.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:02.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:03.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:02.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:03.178 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:02.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:03.178 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:02.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:03.178 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:02.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:03.178 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:02.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:03.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:03.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:03.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:03.176+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:04.044 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:03.714+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27771 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:04.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:03.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:04.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:03.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:04.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:03.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:04.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:03.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:04.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:03.870+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:04.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:03.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:04.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:03.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:04.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:03.871+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:04.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:04.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:04.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:04.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:04.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:04.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:04.202+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:05.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:04.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:05.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:04.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:05.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:04.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:05.177 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:04.844+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:05.177 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:04.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:05.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:04.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:05.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:04.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:05.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:04.893+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:05.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:05.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:05.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:05.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:05.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:05.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:05.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:05.174+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:06.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:05.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:06.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:05.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:06.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:05.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:06.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:05.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:06.164 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:05.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:06.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:05.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:06.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:05.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:06.165 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:05.802+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:06.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:06.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:06.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:06.161+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:07.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:06.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:07.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:06.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:07.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:06.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:07.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:06.837+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:07.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:06.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:07.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:06.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:07.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:06.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:07.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:06.891+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:07.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:07.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:07.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:07.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:07.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:07.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:07.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:07.122+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:08.168 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:07.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:08.168 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:07.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:08.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:07.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:08.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:07.867+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:08.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:07.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:08.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:07.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:08.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:07.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:08.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:07.901+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:08.165+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:08.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:08.165+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:08.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:08.165+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:08.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:08.166+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:09.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:08.718+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27860 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:08.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:08.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:08.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:08.946+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:09.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:08.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:09.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:08.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:09.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:08.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:09.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:08.827+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:09.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:09.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:09.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:09.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:09.132+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:10.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:09.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:10.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:09.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:10.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:09.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:10.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:09.945+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:10.120 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:09.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:10.120 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:09.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:10.120 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:09.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:10.120 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:09.844+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:10.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:10.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:10.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:10.116+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:11.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:10.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:11.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:10.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:11.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:10.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:11.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:10.990+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:11.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:10.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:11.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:10.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:11.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:10.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:11.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:10.857+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:11.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:11.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:11.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:11.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:11.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:11.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:11.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:11.151+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:12.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:12.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:12.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:12.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:12.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:12.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:12.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:12.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:12.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:11.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:12.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:11.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:12.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:11.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:12.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:11.834+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:12.154+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:12.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:12.154+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:12.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:12.154+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:12.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:12.155+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:13.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:12.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:13.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:12.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:13.114 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:12.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:13.114 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:12.851+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:13.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:13.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:13.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:13.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:13.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:13.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:13.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:13.045+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:13.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:13.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:13.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:13.111+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:14.009 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:13.723+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 27948 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:14.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:13.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:14.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:13.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:14.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:13.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:14.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:13.867+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:14.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:14.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:14.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:14.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:14.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:14.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:14.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:14.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:14.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:14.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:14.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:14.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:15.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:15.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:15.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:15.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:15.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:15.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:15.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:15.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:15.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:14.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:15.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:14.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:15.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:14.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:15.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:14.960+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:15.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:14.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:15.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:14.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:15.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:14.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:15.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:14.910+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:16.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:16.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:16.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:16.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:16.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:16.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:16.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:16.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:16.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:15.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:16.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:15.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:16.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:15.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:16.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:15.922+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:16.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:15.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:16.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:15.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:16.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:15.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:16.297 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:15.876+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:17.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:16.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:17.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:16.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:17.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:16.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:17.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:16.913+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:17.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:17.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:17.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:17.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:17.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:17.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:17.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:17.127+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:17.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:16.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:17.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:16.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:17.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:16.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:17.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:16.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:18.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:17.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:18.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:17.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:18.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:17.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:18.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:17.945+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:18.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:18.079+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:18.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:18.079+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:18.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:18.079+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:18.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:18.079+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:18.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:17.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:18.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:17.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:18.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:17.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:18.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:17.908+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:18.995 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:18.727+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28037 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:18.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:18.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:18.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:18.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:18.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:18.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:18.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:18.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:19.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:19.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:19.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:19.081+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:19.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:18.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:19.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:18.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:19.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:18.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:19.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:18.993+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:20.107 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:19.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:20.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:19.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:20.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:19.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:20.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:19.836+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:20.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:20.012+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:20.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:20.012+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:20.109 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:20.012+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:20.109 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:20.013+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:20.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:20.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:20.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:20.105+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:21.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:20.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:21.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:20.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:21.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:20.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:21.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:20.819+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:21.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:21.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:21.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:21.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:21.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:21.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:21.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:21.062+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:21.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:21.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:21.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:21.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:21.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:21.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:21.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:21.007+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:22.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:21.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:22.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:21.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:22.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:21.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:22.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:21.992+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:22.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:22.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:22.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:22.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:22.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:22.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:22.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:22.106+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:22.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:21.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:22.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:21.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:22.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:21.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:22.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:21.860+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:23.080 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:22.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:23.080 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:22.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:23.080 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:22.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:23.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:22.813+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:23.081 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:22.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:23.081 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:22.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:23.081 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:22.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:23.081 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:22.952+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:23.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:23.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:23.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:23.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:23.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:23.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:23.747 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:23.078+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:24.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:23.732+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28126 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:24.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:23.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:24.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:23.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:24.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:23.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:24.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:23.976+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:24.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:23.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:24.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:23.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:24.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:23.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:24.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:23.783+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:24.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:24.097+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:24.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:24.097+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:24.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:24.097+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:24.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:24.098+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:25.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:24.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:25.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:24.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:25.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:24.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:25.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:24.751+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:25.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:24.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:25.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:24.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:25.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:24.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:25.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:24.948+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:25.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:25.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:25.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:25.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:25.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:25.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:25.124+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:26.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:25.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:26.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:25.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:26.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:25.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:26.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:25.928+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:25.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:25.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:26.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:25.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:26.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:25.734+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:26.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:26.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:26.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:26.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:26.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:26.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:26.168+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:26.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:27.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:26.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:27.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:26.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:27.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:26.933+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:27.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:26.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:26.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:26.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:26.752+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:27.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:27.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:27.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:27.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:27.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:27.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:27.129+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:27.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:28.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:27.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:28.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:27.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:28.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:27.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:28.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:27.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:28.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:27.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:28.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:27.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:28.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:27.726+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:28.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:28.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:28.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:28.103+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:28.992 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:28.736+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28215 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:28.992 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:28.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:28.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:28.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:28.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:28.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:28.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:28.762+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:29.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:29.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:29.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:29.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:29.059+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:29.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:28.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:29.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:28.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:29.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:28.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:29.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:28.990+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:30.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:29.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:30.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:29.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:30.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:29.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:30.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:29.753+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:30.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:30.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:30.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:30.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:30.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:30.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:30.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:30.031+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:30.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:30.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:30.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:30.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:30.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:30.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:30.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:30.037+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:31.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:31.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:31.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:31.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:31.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:31.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:31.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:31.017+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:31.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:30.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:31.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:30.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:31.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:30.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:31.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:30.767+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:31.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:31.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:31.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:31.052+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:32.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:32.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:32.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:32.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:32.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:32.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:32.033+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:32.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:31.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:32.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:31.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:32.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:31.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:32.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:31.785+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:32.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:32.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:32.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:32.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:32.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:32.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:32.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:32.010+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:33.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:32.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:33.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:32.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:33.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:32.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:33.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:32.775+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:33.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:33.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:33.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:33.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:33.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:33.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:33.030+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:33.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:33.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:33.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:33.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:33.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:33.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:33.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:33.045+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:34.019 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:33.741+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28305 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:33.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:33.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:33.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:33.764+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:34.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:34.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:34.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:34.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:34.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:34.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:34.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:34.017+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:34.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:34.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:34.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:34.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:34.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:34.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:34.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:34.073+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:35.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:35.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:35.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:35.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:35.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:35.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:35.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:35.009+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:35.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:34.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:35.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:34.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:35.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:34.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:35.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:34.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:35.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:35.054+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:35.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:35.054+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:35.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:35.054+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:35.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:35.055+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:36.084 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:36.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:36.084 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:36.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:36.084 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:36.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:36.084 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:36.035+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:36.084 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:35.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:36.084 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:35.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:36.085 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:35.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:36.085 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:35.826+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:36.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:36.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:36.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:36.081+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:36.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:36.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:37.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:36.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:37.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:36.778+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:37.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:37.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:37.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:37.081+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:37.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:37.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:37.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:37.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:37.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:37.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:37.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:37.119+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:37.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:38.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:37.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:38.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:37.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:38.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:37.758+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:38.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:38.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:38.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:38.055+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:38.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:38.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:38.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:38.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:38.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:38.162+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:39.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:38.746+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28394 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:39.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:38.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:39.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:38.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:39.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:38.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:39.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:38.723+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:39.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:39.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:39.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:39.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:39.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:39.053+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:39.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:39.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:39.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:39.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:39.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:39.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:39.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:39.152+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:39.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:39.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:39.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:39.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:39.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:39.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:39.705+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:40.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:40.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:40.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:40.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:40.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:40.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:40.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:40.045+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:40.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:40.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:40.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:40.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:40.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:40.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:40.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:40.114+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:40.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:41.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:40.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:41.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:40.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:41.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:40.672+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:41.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:41.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:41.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:41.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:41.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:41.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:41.118+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:41.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:41.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:41.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:41.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:41.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:41.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:41.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:41.058+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:42.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:41.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:42.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:41.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:42.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:41.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:42.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:41.660+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:42.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:42.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:42.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:42.114+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:42.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:42.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:42.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:42.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:42.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:42.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:42.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:42.031+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:43.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:42.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:43.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:42.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:43.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:42.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:43.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:42.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:43.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:43.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:43.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:43.145+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:43.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:43.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:43.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:43.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:43.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:43.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:43.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:43.009+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:44.030 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:43.751+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28483 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:44.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:43.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:44.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:43.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:44.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:43.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:44.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:43.647+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:44.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:44.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:44.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:44.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:44.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:44.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:44.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:44.170+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:44.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:44.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:44.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:44.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:44.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:44.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:44.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:44.028+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:45.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:44.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:45.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:44.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:45.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:44.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:45.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:44.634+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:45.315 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:45.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:45.315 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:45.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:45.315 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:45.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:45.315 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:45.139+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:45.316 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:45.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:45.316 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:45.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:45.316 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:45.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:45.316 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:45.054+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:46.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:45.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:46.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:45.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:46.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:45.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:46.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:45.586+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:46.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:46.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:46.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:46.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:46.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:46.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:46.015+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:46.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:46.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:46.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:46.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:46.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:46.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:46.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:46.147+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:46.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:46.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:46.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:46.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:46.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:46.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:46.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:46.570+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:47.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:47.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:47.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:47.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:47.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:47.149+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:47.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:46.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:47.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:46.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:47.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:46.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:47.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:46.983+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:47.984 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:47.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:47.984 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:47.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:47.984 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:47.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:47.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:47.581+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:48.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:48.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:48.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:48.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:48.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:48.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:48.152+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:47.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:47.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:47.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:48.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:47.982+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:48.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:48.755+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28572 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:48.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:48.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:48.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:48.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:48.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:48.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:48.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:48.606+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:49.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:48.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:49.150 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:48.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:49.150 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:48.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:49.150 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:48.997+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:49.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:49.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:49.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:49.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:49.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:49.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:49.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:49.147+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:49.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:49.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:49.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:49.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:49.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:49.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:49.643+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:50.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:50.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:50.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:50.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:50.003+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:50.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:50.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:50.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:50.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:50.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:50.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:50.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:50.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:51.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:50.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:51.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:50.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:51.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:50.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:51.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:50.597+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:51.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:51.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:51.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:51.152+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:51.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:51.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:51.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:51.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:51.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:51.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:51.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:51.006+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:51.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:51.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:51.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:51.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:51.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:51.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:51.592+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:51.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:52.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:51.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:52.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:51.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:52.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:51.987+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:52.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:52.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:52.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:52.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:52.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:52.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:52.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:52.147+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:53.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:52.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:53.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:52.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:53.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:52.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:53.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:52.544+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:53.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:53.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:53.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:53.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:53.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:53.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:53.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:53.033+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:53.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:53.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:53.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:53.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:53.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:53.155+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:53.888 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:53.760+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28662 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:53.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:53.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:53.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:53.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:53.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:53.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:53.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:53.577+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:54.003+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:54.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:54.003+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:54.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:54.003+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:54.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:54.003+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:54.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:54.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:54.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:54.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:54.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:54.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:54.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:54.107+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:54.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:54.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:54.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:54.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:54.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:54.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:54.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:54.618+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:54.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:54.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:55.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:54.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:55.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:54.983+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:55.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:55.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:55.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:55.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:55.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:55.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:55.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:55.117+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:56.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:55.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:56.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:55.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:56.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:55.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:56.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:55.614+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:56.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:56.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:56.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:56.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:56.031+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:56.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:56.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:56.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:56.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:56.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:56.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:56.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:56.103+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:57.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:56.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:57.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:56.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:57.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:56.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:57.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:56.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:57.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:57.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:57.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:57.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:57.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:57.041+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:57.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:57.042+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:57.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:57.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:57.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:57.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:57.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:57.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:57.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:57.154+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:58.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:57.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:58.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:57.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:58.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:57.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:58.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:57.635+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:58.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:58.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:58.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:58.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:58.030+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:58.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:58.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:58.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:58.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:58.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:58.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:58.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:58.107+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:58.892 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:33:58.765+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28751 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:33:58.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:58.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:58.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:58.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:58.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:58.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:58.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:58.601+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:33:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:59.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:59.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:59.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:33:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:33:59.023+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:59.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:59.077+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:59.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:59.077+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:59.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:59.077+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:33:59.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:33:59.077+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:33:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:59.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:59.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:59.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:59.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:59.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:33:59.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:33:59.650+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:00.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:00.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:00.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:00.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:00.013+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:00.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:00.013+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:00.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:00.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:00.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:00.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:00.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:00.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:00.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:00.040+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:00.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:00.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:00.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:00.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:00.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:00.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:00.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:00.669+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:00.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:01.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:00.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:01.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:00.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:01.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:00.976+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:01.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:01.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:01.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:01.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:01.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:01.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:01.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:01.004+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:02.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:01.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:02.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:01.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:02.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:01.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:02.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:01.643+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:02.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:02.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:02.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:02.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:02.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:02.009+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:02.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:02.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:02.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:02.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:02.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:02.051+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:02.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:02.051+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:03.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:02.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:03.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:02.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:03.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:02.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:03.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:02.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:03.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:03.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:03.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:03.055+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:03.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:03.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:03.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:03.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:03.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:03.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:03.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:03.080+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:04.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:03.770+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28840 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:03.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:03.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:03.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:03.712+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:04.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:04.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:04.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:04.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:04.102+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:04.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:04.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:04.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:04.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:04.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:04.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:04.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:04.040+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:04.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:04.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:05.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:04.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:05.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:04.738+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:05.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:05.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:05.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:05.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:05.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:05.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:05.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:05.092+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:05.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:05.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:05.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:05.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:05.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:05.064+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:05.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:05.065+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:05.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:05.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:05.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:06.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:05.708+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:06.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:06.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:06.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:06.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:06.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:06.100+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:06.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:06.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:06.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:06.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:06.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:06.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:06.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:06.101+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:07.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:06.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:07.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:06.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:07.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:06.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:07.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:06.745+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:07.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:07.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:07.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:07.119+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:07.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:07.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:07.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:07.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:07.140+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:07.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:07.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:07.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:08.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:07.712+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:08.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:08.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:08.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:08.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:08.081+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:08.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:08.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:08.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:08.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:08.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:08.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:08.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:08.117+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:09.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:08.775+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 28929 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:08.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:08.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:08.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:08.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:09.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:09.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:09.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:09.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:09.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:09.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:09.078+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:09.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:09.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:09.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:09.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:09.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:09.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:09.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:09.093+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:09.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:09.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:09.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:10.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:09.757+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:10.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:10.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:10.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:10.120+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:10.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:10.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:10.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:10.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:10.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:10.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:10.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:10.135+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:10.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:10.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:10.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:10.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:10.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:10.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:10.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:10.743+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:11.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:11.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:11.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:11.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:11.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:11.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:11.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:11.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:11.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:11.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:11.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:11.174+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:11.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:12.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:11.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:12.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:11.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:12.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:11.763+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:12.209+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:12.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:12.209+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:12.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:12.209+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:12.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:12.209+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:12.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:12.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:12.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:12.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:12.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:12.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:12.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:12.160+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:12.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:13.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:12.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:13.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:12.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:13.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:12.768+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:13.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:13.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:13.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:13.162+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:13.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:13.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:13.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:13.141+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:14.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:13.780+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29018 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:14.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:13.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:14.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:13.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:14.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:13.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:14.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:13.728+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:14.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:14.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:14.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:14.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:14.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:14.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:14.177+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:14.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:14.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:14.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:14.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:14.168+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:14.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:14.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:14.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:14.709+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:15.172+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:15.172+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:15.172+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:15.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:15.172+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:15.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:15.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:15.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:15.128+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:15.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:15.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:15.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:16.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:15.698+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:16.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:16.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:16.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:16.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:16.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:16.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:16.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:16.190+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:16.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:16.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:16.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:16.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:16.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:16.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:16.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:16.128+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:16.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:16.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:16.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:17.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:16.736+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:17.193+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:17.193+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:17.193+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:17.193+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:17.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:17.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:17.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:17.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:17.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:17.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:17.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:17.131+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:17.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:17.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:17.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:18.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:17.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:18.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:18.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:18.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:18.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:18.207+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:18.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:18.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:18.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:18.170+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:19.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:18.785+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29106 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:19.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:18.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:19.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:18.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:19.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:18.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:19.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:18.774+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:19.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:19.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:19.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:19.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:19.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:19.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:19.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:19.207+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:19.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:19.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:19.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:19.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:19.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:19.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:19.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:19.235+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:20.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:19.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:20.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:19.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:20.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:19.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:20.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:19.805+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:20.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:20.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:20.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:20.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:20.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:20.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:20.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:20.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:20.186+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:21.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:20.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:21.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:20.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:21.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:20.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:21.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:20.827+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:21.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:21.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:21.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:21.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:21.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:21.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:21.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:21.259+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:21.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:21.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:21.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:21.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:21.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:21.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:21.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:21.157+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:22.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:21.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:22.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:21.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:22.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:21.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:22.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:21.864+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:22.175+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:22.175+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:22.175+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:22.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:22.176+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:22.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:22.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:22.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:22.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:22.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:22.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:22.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:22.259+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:23.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:22.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:23.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:22.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:23.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:22.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:23.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:22.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:23.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:23.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:23.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:23.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:23.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:23.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:23.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:23.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:23.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:23.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:23.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:23.155+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:24.043 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:23.789+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29186 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:23.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:24.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:23.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:24.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:23.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:24.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:23.903+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:24.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:24.316+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:24.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:24.316+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:24.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:24.316+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:24.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:24.316+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:24.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:24.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:24.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:24.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:24.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:24.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:24.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:24.116+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:25.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:25.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:25.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:25.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:25.068+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:25.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:24.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:25.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:24.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:25.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:24.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:25.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:24.855+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:25.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:25.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:25.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:25.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:25.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:26.264 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:25.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:26.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:25.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:26.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:25.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:26.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:25.861+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:26.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:26.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:26.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:26.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:26.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:26.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:26.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:26.061+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:26.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:26.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:26.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:26.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:26.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:26.262+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:26.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:27.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:26.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:27.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:26.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:27.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:26.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:27.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:27.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:27.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:27.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:27.096+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:27.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:27.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:27.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:27.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:27.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:27.297+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:28.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:27.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:28.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:27.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:28.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:27.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:28.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:27.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:28.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:28.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:28.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:28.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:28.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:28.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:28.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:28.100+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:28.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:28.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:28.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:28.284+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:29.097 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:28.794+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29274 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:29.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:28.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:29.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:28.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:29.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:28.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:29.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:28.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:29.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:29.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:29.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:29.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:29.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:29.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:29.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:29.274+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:29.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:29.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:29.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:29.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:29.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:29.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:29.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:29.096+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:30.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:29.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:30.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:29.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:30.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:29.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:30.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:29.891+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:30.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:30.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:30.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:30.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:30.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:30.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:30.214 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:30.097+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:30.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:30.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:30.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:30.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:30.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:30.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:30.310+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:30.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:30.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:31.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:30.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:31.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:30.883+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:31.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:31.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:31.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:31.062+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:31.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:31.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:31.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:31.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:31.301+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:32.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:31.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:32.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:31.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:32.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:31.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:32.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:31.896+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:32.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:32.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:32.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:32.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:32.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:32.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:32.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:32.089+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:32.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:32.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:32.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:32.278+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:32.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:32.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:32.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:33.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:32.937+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:33.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:33.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:33.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:33.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:33.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:33.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:33.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:33.088+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:33.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:33.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:33.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:33.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:33.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:33.320+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:34.114 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:33.799+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29364 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:34.115 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:33.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:34.115 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:33.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:34.115 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:33.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:34.115 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:33.982+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:34.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:34.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:34.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:34.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:34.346+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:34.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:34.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:34.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:34.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:34.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:34.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:34.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:34.112+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:35.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:35.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:35.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:35.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:35.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:35.113+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:35.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:34.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:35.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:34.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:35.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:34.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:35.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:34.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:35.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:35.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:35.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:35.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:35.310+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:36.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:36.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:36.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:36.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:36.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:36.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:36.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:36.144+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:36.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:35.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:36.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:35.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:36.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:35.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:36.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:35.945+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:36.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:36.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:36.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:36.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:36.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:37.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:37.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:37.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:37.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:37.139+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:37.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:36.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:37.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:36.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:37.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:36.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:37.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:36.916+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:37.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:37.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:37.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:37.333+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:38.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:38.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:38.148+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:38.148+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:38.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:37.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:38.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:37.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:38.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:37.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:38.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:37.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:38.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:38.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:38.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:38.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:38.292+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:39.117 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:38.805+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29453 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:39.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:38.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:39.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:38.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:39.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:38.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:39.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:38.888+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:39.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:39.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:39.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:39.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:39.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:39.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:39.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:39.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:39.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:39.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:39.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:39.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:39.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:40.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:39.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:40.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:39.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:40.213 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:39.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:40.213 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:39.880+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:40.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:40.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:40.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:40.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:40.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:40.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:40.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:40.070+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:40.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:40.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:40.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:40.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:40.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:40.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:40.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:40.288+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:41.263 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:41.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:41.264 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:41.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:41.264 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:41.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:41.264 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:41.076+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:41.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:40.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:41.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:40.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:41.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:40.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:41.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:40.873+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:41.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:41.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:41.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:41.261+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:42.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:42.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:42.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:42.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:42.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:42.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:42.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:42.093+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:42.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:41.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:42.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:41.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:42.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:41.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:42.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:41.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:42.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:42.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:42.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:42.226+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:43.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:42.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:43.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:42.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:43.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:42.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:43.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:42.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:43.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:43.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:43.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:43.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:43.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:43.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:43.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:43.121+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:43.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:43.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:43.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:43.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:43.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:43.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:43.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:43.263+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:44.127 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:43.809+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29542 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:44.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:43.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:44.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:43.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:44.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:43.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:44.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:43.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:44.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:44.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:44.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:44.276+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:44.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:44.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:44.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:44.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:44.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:44.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:44.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:44.125+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:45.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:45.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:45.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:45.090+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:45.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:44.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:45.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:44.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:45.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:44.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:45.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:44.910+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:45.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:45.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:45.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:45.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:46.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:46.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:46.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:46.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:46.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:46.086+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:46.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:45.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:46.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:45.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:46.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:45.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:46.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:45.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:46.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:46.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:46.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:46.361+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:47.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:47.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:47.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:47.039+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:46.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:46.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:46.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:46.924+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:47.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:47.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:47.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:47.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:47.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:47.324+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:48.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:48.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:48.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:48.074+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:48.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:47.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:48.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:47.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:48.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:47.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:48.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:47.967+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:48.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:48.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:48.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:48.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:49.099 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:48.815+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29631 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:49.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:48.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:49.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:48.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:49.100 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:48.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:49.100 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:48.973+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:49.354 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:49.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:49.355 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:49.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:49.355 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:49.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:49.355 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:49.097+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:49.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:49.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:49.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:49.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:50.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:50.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:50.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:50.016+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:50.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:50.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:50.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:50.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:50.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:50.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:50.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:50.132+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:50.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:50.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:50.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:50.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:50.338+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:51.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:51.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:51.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:51.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:51.025+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:51.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:51.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:51.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:51.106+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:51.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:51.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:51.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:51.383+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:52.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:52.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:52.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:52.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:52.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:52.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:52.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:52.080+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:52.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:52.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:52.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:52.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:52.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:52.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:52.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:52.066+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:52.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:52.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:52.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:52.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:53.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:53.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:53.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:53.041+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:53.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:53.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:53.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:53.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:53.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:53.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:53.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:53.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:53.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:53.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:53.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:53.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:54.079 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:53.819+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29721 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:54.079 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:54.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:54.080 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:54.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:54.080 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:54.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:54.080 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:54.019+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:54.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:54.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:54.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:54.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:54.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:54.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:54.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:54.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:54.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:54.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:54.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:54.335+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:55.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:55.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:55.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:55.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:55.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:55.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:55.091+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:55.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:55.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:55.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:55.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:55.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:55.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:55.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:55.005+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:55.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:55.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:55.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:55.355+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:56.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:56.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:56.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:56.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:56.060+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:56.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:56.036+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:56.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:56.036+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:56.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:56.036+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:56.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:56.037+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:56.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:56.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:56.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:56.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:56.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:56.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:56.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:57.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:57.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:57.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:57.074+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:57.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:57.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:57.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:57.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:57.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:57.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:57.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:57.029+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:57.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:57.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:57.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:57.439+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:58.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:58.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:58.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:58.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:58.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:58.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:58.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:58.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:58.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:58.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:58.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:58.024+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:58.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:58.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:58.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:58.471+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:59.114 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:34:58.824+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29809 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:34:59.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:59.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:59.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:59.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:59.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:59.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:34:59.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:34:59.060+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:34:59.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:59.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:59.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:59.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:59.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:59.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:34:59.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:34:59.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:34:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:59.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:59.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:59.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:34:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:34:59.493+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:00.469 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:00.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:00.469 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:00.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:00.470 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:00.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:00.470 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:00.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:00.470 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:00.084+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:00.470 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:00.084+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:00.470 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:00.084+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:00.470 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:00.084+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:00.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:00.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:00.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:00.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:01.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:01.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:01.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:01.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:01.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:01.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:01.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:01.148+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:01.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:01.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:01.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:01.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:01.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:01.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:01.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:01.108+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:01.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:01.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:01.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:01.500+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:02.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:02.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:02.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:02.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:02.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:02.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:02.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:02.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:02.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:02.066+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:02.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:02.066+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:02.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:02.066+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:02.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:02.066+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:02.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:02.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:02.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:02.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:03.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:03.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:03.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:03.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:03.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:03.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:03.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:03.119+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:03.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:03.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:03.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:03.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:03.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:03.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:03.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:03.094+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:03.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:03.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:03.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:03.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:04.102 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:03.829+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29898 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:04.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:04.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:04.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:04.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:04.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:04.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:04.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:04.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:04.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:04.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:04.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:04.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:04.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:04.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:04.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:04.100+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:04.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:04.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:04.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:04.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:04.489+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:05.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:05.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:05.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:05.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:05.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:05.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:05.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:05.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:05.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:05.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:05.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:05.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:05.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:05.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:05.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:05.056+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:05.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:05.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:05.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:05.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:06.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:06.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:06.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:06.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:06.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:06.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:06.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:06.102+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:06.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:06.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:06.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:06.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:06.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:06.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:06.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:06.047+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:06.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:06.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:06.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:06.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:06.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:07.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:07.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:07.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:07.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:07.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:07.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:07.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:07.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:07.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:07.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:07.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:07.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:07.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:07.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:07.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:07.076+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:07.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:07.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:07.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:07.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:07.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:07.495+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:08.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:08.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:08.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:08.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:08.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:08.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:08.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:08.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:08.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:08.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:08.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:08.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:08.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:08.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:08.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:08.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:08.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:08.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:08.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:08.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:09.154 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:08.835+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 29986 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:09.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:09.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:09.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:09.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:09.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:09.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:09.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:09.158+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:09.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:09.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:09.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:09.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:09.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:09.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:09.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:09.153+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:09.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:09.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:09.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:09.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:10.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:10.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:10.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:10.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:10.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:10.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:10.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:10.125+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:10.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:10.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:10.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:10.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:10.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:10.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:10.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:10.143+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:10.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:10.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:10.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:10.535+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:11.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:11.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:11.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:11.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:11.131+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:11.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:11.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:11.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:11.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:11.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:11.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:11.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:11.158+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:11.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:11.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:11.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:11.519+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:12.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:12.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:12.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:12.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:12.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:12.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:12.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:12.135+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:12.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:12.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:12.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:12.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:12.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:12.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:12.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:12.205+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:12.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:12.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:12.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:12.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:13.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:13.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:13.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:13.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:13.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:13.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:13.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:13.125+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:13.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:13.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:13.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:13.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:13.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:13.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:13.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:13.238+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:13.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:13.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:13.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:13.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:13.527+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:14.131 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:13.839+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30076 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:14.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:14.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:14.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:14.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:14.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:14.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:14.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:14.222+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:14.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:14.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:14.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:14.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:14.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:14.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:14.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:14.129+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:14.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:14.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:14.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:14.520+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:15.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:15.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:15.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:15.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:15.171+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:15.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:15.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:15.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:15.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:15.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:15.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:15.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:16.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:15.559+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:16.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:16.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:16.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:16.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:16.122+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:16.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:16.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:16.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:16.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:16.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:16.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:16.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:16.254+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:16.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:16.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:16.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:16.585+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:17.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:17.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:17.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:17.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:17.116+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:17.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:17.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:17.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:17.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:17.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:17.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:17.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:17.297+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:17.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:17.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:18.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:17.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:18.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:17.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:18.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:18.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:18.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:18.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:18.089+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:18.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:18.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:18.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:18.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:18.847 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:18.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:18.847 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:18.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:18.847 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:18.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:18.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:18.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:19.122 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:18.844+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30165 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:19.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:19.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:19.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:19.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:19.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:19.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:19.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:19.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:19.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:19.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:19.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:19.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:19.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:19.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:19.252+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:19.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:19.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:19.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:19.582+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:20.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:20.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:20.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:20.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:20.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:20.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:20.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:20.127+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:20.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:20.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:20.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:20.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:20.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:20.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:20.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:20.267+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:20.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:20.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:20.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:20.598+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:21.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:21.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:21.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:21.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:21.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:21.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:21.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:21.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:21.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:21.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:21.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:21.306+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:21.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:21.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:21.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:21.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:22.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:22.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:22.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:22.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:22.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:22.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:22.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:22.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:22.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:22.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:22.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:22.295+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:22.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:22.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:22.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:22.607+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:23.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:23.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:23.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:23.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:23.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:23.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:23.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:23.255+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:23.865 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:23.850+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30254 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:23.865 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:23.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:23.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:23.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:23.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:23.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:23.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:23.610+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:24.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:24.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:24.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:24.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:24.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:24.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:24.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:24.027+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:24.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:24.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:24.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:24.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:24.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:24.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:24.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:24.252+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:25.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:24.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:25.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:24.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:25.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:24.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:25.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:24.616+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:25.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:25.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:25.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:25.016+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:25.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:25.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:25.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:25.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:25.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:25.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:25.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:25.253+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:26.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:25.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:26.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:25.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:26.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:25.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.576246+0000 (oldest deadline 2024-02-19T19:25:14.576246+0000) 2024-02-19T19:35:26.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:25.613+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:26.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:26.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:26.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:26.007+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:26.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:26.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:26.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:26.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:26.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:26.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:26.233+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:26.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:26.654+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:26.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:26.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:26.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:26.974+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:27.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:27.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:27.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:24:54.059675+0000 (oldest deadline 2024-02-19T19:25:14.059675+0000) 2024-02-19T19:35:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:27.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:27.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:27.684+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:28.191+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:27.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:27.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:27.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:28.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:27.968+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:28.975 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:28.855+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30343 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:28.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:28.722+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:29.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:29.155+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:28.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:28.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:28.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:28.973+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:29.674+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:30.135+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:30.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:30.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:30.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:30.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:30.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:30.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:30.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:30.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:30.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:30.688+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:31.166+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:30.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:30.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:30.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:25:01.937125+0000 (oldest deadline 2024-02-19T19:25:21.937125+0000) 2024-02-19T19:35:31.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:30.982+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:31.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:31.672+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:32.147+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:31.994+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:33.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:32.644+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:33.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:33.156+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:33.007+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:33.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:33.860+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30433 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:33.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:33.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:34.049+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:34.181+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:34.663+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:35.079+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:35.136+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:35.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:36.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:36.102+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:36.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:37.069+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:37.124+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:37.701+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:38.042+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:38.145+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:38.992 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:38.865+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30522 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:38.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:38.693+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:39.055+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:39.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:39.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:39.735+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:40.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:40.073+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:40.137+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:40.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:41.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:41.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:41.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:41.096+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:42.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:41.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:42.139+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:42.051+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:42.695+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:43.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:43.046+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:43.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:43.113+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:43.998 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:43.871+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30611 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:43.998 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:43.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:44.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:44.110+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:43.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:44.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:44.674+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:45.100+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:44.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:45.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:45.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:46.052+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:45.913+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:46.695+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:47.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:46.910+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:47.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:47.046+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:47.692+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:48.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:47.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:48.068+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:49.005 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:48.876+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30700 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:49.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:48.892+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:49.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:48.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:49.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:48.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:49.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:48.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:49.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:48.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:49.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:49.110+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:49.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:49.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:49.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:49.740+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:50.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:50.156+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:50.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:49.927+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:50.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:50.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:50.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:50.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:50.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:50.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:50.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:50.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:51.250 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:51.196+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:51.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:50.974+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:51.990 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:51.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:51.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:51.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:51.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:51.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:51.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:51.697+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:51.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:52.229+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:52.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:52.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:52.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:52.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:52.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:52.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:52.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:52.696+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:53.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:52.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:53.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:53.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:53.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:53.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:53.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:53.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:53.268+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:54.008 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:53.881+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30790 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:54.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:53.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:54.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:53.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:54.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:53.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:54.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:53.703+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:54.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:53.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:54.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:53.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:54.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:53.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:54.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:53.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:54.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:54.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:54.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:54.256+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:54.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:54.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:54.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:54.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:54.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:55.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:54.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:55.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:54.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:55.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:54.921+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:55.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:55.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:55.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:55.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:55.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:55.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:55.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:55.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:55.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:55.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:55.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:55.694+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:56.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:55.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:56.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:55.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:56.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:55.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:56.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:55.889+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:56.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:56.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:56.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:56.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:56.249+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:56.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:56.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:56.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:56.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:56.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:56.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:56.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:56.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:57.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:56.932+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:57.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:56.932+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:57.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:56.932+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:57.246 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:56.933+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:57.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:57.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:57.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:57.244+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:57.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:57.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:57.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:57.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:57.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:57.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:57.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:57.646+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:58.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:57.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:58.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:57.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:58.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:57.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:58.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:57.952+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:58.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:58.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:58.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:58.252+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:58.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:58.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:58.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:58.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:58.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:58.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:58.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:58.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:59.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:35:58.886+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30879 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:35:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:58.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:58.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:58.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:35:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:58.944+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:35:59.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:59.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:59.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:59.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:59.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:59.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:35:59.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:35:59.290+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:35:59.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:59.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:59.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:59.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:59.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:59.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:35:59.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:35:59.639+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:00.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:59.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:00.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:59.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:00.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:59.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:00.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:35:59.958+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:00.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:00.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:00.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:00.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:00.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:00.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:00.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:00.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:00.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:00.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:00.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:00.616+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:00.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:00.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:00.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:00.940+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:01.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:01.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:01.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:01.350+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:01.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:01.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:01.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:01.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:01.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:01.580+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:01.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:01.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:01.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:01.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:02.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:02.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:02.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:02.364+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:02.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:02.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:02.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:02.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:02.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:02.597+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:02.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:02.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:02.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:02.876+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:03.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:03.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:03.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:03.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:03.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:03.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:03.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:03.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:03.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:03.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:03.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:03.598+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:04.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:03.891+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 30968 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:03.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:03.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:03.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:03.829+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:04.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:04.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:04.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:04.563+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:04.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:04.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:04.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:04.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:04.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:04.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:04.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:04.418+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:04.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:04.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:04.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:04.855+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:05.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:05.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:05.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:05.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:05.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:05.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:05.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:05.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:05.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:05.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:05.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:05.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:05.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:05.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:05.862+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:06.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:06.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:06.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:06.574+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:06.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:06.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:06.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:06.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:06.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:06.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:06.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:06.402+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:06.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:06.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:06.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:06.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:07.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:07.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:07.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:07.526+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:07.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:07.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:07.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:07.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:07.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:07.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:07.390+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:07.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:07.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:07.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:07.810+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:08.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:08.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:08.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:08.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:08.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:08.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:08.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:08.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:08.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:08.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:08.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:08.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:08.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:08.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:08.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:08.375+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:09.023 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:08.897+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31057 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:09.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:08.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:09.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:08.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:09.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:08.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:09.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:08.767+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:09.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:09.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:09.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:09.425+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:09.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:09.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:09.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:09.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:09.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:09.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:09.549+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:10.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:09.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:10.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:09.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:10.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:09.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:10.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:09.792+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:10.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:10.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:10.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:10.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:10.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:10.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:10.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:10.408+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:10.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:10.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:10.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:10.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:10.501+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:10.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:10.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:10.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:10.780+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:11.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:11.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:11.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:11.508+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:11.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:11.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:11.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:11.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:11.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:11.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:11.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:11.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:11.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:11.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:11.818+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:12.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:12.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:12.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:12.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:12.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:12.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:12.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:12.408+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:12.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:12.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:12.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:12.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:12.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:12.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:12.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:12.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:12.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:12.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:12.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:12.817+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:13.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:13.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:13.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:13.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:13.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:13.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:13.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:13.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:13.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:13.366+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:14.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:13.902+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31146 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:13.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:13.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:13.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:13.864+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:14.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:14.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:14.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:14.457+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:14.368+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:14.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:14.368+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:14.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:14.368+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:14.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:14.368+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:14.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:14.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:14.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:14.826+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:15.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:15.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:15.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:15.452+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:15.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:15.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:15.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:15.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:15.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:15.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:15.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:15.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:15.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:15.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:15.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:15.835+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:16.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:16.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:16.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:16.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:16.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:16.406+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:16.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:16.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:16.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:16.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:16.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:16.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:16.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:16.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:16.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:16.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:16.865+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:17.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:17.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:17.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:17.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:17.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:17.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:17.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:17.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:17.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:17.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:18.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:17.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:18.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:17.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:18.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:18.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:18.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:18.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:18.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:18.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:18.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:18.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:18.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:18.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:18.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:18.291+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:19.272 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:18.907+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31236 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:19.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:18.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:19.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:18.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:19.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:18.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:19.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:18.925+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:19.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:19.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:19.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:19.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:19.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:19.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:19.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:19.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:19.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:19.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:19.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:19.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:19.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:19.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:19.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:19.270+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:20.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:19.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:20.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:19.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:20.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:19.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:20.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:19.897+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:20.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:20.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:20.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:20.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:20.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:20.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:20.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:20.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:20.259+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:20.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:20.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:20.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:20.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:21.386+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:21.386+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:21.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:21.387+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:21.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:21.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:21.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:21.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:21.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:21.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:21.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:21.294+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:22.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:21.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:22.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:21.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:22.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:21.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:22.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:21.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:22.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:22.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:22.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:22.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:22.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:22.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:22.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:22.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:22.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:22.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:22.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:22.300+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:22.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:22.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:22.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:22.841+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:23.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:23.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:23.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:23.307+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:23.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:23.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:23.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:23.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:23.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:23.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:24.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:23.913+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31324 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:23.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:23.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:23.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:23.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:24.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:24.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:24.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:24.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:24.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:24.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:24.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:24.307+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:24.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:24.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:24.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:24.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:25.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:25.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:25.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:25.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:25.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:25.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:25.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:25.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:25.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:26.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:25.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:26.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:25.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:26.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:25.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:26.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:25.863+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:26.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:26.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:26.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:26.375+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:26.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:26.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:26.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:26.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:26.260+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:27.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:26.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:27.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:26.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:27.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:26.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:27.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:26.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:27.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:27.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:27.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:27.369+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:27.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:27.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:27.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:27.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:28.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:27.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:28.249 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:27.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:28.249 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:27.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:28.249 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:27.911+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:28.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:28.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:28.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:28.247+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:28.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:28.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:28.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:28.335+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:29.219 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:28.918+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31414 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:29.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:28.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:29.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:28.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:29.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:28.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:29.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:28.923+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:29.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:29.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:29.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:29.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:29.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:29.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:29.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:29.218+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:29.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:29.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:29.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:29.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:29.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:29.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:29.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:29.383+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:30.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:29.952+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:30.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:29.952+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:30.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:29.952+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:30.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:29.953+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:30.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:30.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:30.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:30.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:30.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:30.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:30.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:30.171+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:30.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:30.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:30.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:30.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:30.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:30.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:30.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:30.968+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:31.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:31.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:31.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:31.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:31.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:31.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:31.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:31.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:31.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:32.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:32.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:32.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:32.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:32.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:32.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:32.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:32.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:32.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:32.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:32.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:32.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:33.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:32.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:33.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:32.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:33.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:32.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:33.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:32.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:33.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:33.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:33.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:33.437+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:33.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:33.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:33.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:33.263+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:34.281 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:33.924+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31503 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:34.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:33.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:34.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:33.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:34.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:33.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:34.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:33.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:34.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:34.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:34.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:34.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:34.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:34.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:34.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:34.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:34.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:34.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:34.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:35.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:34.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:35.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:34.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:35.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:34.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:35.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:34.990+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:35.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:35.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:35.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:35.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:35.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:35.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:35.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:35.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:35.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:35.250+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:36.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:35.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:36.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:35.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:36.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:35.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:36.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:35.972+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:36.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:36.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:36.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:36.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:36.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:36.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:36.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:36.248+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:37.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:36.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:37.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:36.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:37.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:36.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:37.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:36.964+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:37.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:37.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:37.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:37.393+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:37.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:37.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:37.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:37.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:37.277+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:38.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:37.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:38.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:37.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:38.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:37.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:38.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:37.922+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:38.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:38.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:38.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:38.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:38.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:38.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:38.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:38.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:38.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:39.194 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:38.929+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31593 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:39.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:38.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:39.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:38.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:39.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:38.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:39.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:38.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:39.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:39.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:39.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:39.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:39.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:39.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:39.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:39.356+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:39.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:39.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:39.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:39.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:39.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:39.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:39.193+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:40.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:39.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:40.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:39.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:40.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:39.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:40.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:39.948+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:40.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:40.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:40.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:40.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:40.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:40.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:40.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:40.382+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:40.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:40.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:40.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:40.207+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:41.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:41.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:41.233+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:41.233+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:40.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:41.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:40.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:41.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:40.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:41.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:40.993+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:41.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:41.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:41.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:41.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:41.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:41.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:41.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:41.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:41.990+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:42.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:42.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:42.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:42.213+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:42.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:42.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:42.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:42.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:43.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:43.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:43.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:43.028+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:43.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:43.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:43.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:43.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:43.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:43.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:43.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:43.377+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:44.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:43.935+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31680 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:44.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:44.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:44.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:44.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:44.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:44.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:44.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:44.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:44.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:44.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:44.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:44.236+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:44.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:44.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:44.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:44.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:44.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:44.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:44.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:44.390+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:45.396 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:45.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:45.396 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:45.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:45.396 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:45.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:45.396 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:45.104+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:45.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:45.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:45.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:45.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:45.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:45.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:45.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:45.249+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:45.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:45.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:45.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:45.395+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:46.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:46.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:46.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:46.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:46.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:46.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:46.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:46.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:46.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:46.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:46.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:46.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:46.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:46.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:46.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:46.122+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:46.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:46.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:46.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:46.415+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:47.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:47.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:47.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:47.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:47.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:47.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:47.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:47.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:47.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:47.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:47.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:47.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:47.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:47.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:47.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:47.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:47.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:47.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:47.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:47.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:48.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:48.292+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:48.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:48.292+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:48.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:48.292+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:48.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:48.293+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:48.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:48.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:48.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:48.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:48.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:48.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:48.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:48.066+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:48.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:48.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:48.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:48.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:49.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:48.940+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31769 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:49.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:49.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:49.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:49.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:49.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:49.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:49.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:49.429+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:49.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:49.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:49.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:49.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:50.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:50.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:50.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:50.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:50.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:50.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:50.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:50.393+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:50.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:50.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:50.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:50.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:50.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:50.304+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:51.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:51.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:51.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:51.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:51.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:51.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:51.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:51.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:51.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:51.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:51.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:51.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:51.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:52.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:52.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:52.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:52.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:52.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:52.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:52.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:52.301+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:52.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:52.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:52.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:52.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:52.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:52.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:52.314 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:52.055+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:52.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:52.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:52.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:52.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:52.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:52.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:52.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:52.312+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:53.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:53.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:53.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:53.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:53.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:53.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:53.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:53.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:53.336+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:53.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:53.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:53.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:53.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:53.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:53.315+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:54.293 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:53.946+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31858 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:54.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:54.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:54.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:54.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:54.382+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:54.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:54.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:54.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:54.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:54.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:54.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:54.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:55.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:55.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:55.355 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:55.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:55.355 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:55.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:55.355 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:55.068+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:55.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:55.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:55.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:55.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:55.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:55.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:55.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:55.353+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:56.352 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:56.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:56.353 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:56.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:56.353 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:56.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:56.353 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:56.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:56.353 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:56.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:56.353 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:56.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:56.353 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:56.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:56.353 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:56.076+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:56.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:56.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:56.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:56.351+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:57.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:57.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:57.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:57.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:57.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:57.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:57.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:57.291+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:57.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:57.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:57.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:57.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:57.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:57.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:57.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:57.093+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:57.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:57.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:57.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:57.400+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:58.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:58.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:58.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:58.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:58.433 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:58.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:58.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:58.303+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:58.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:58.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:58.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:58.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:58.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:58.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:58.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:58.067+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:58.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:58.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:58.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:58.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:59.288 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:36:58.952+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 31948 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:36:59.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:59.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:59.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:59.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:59.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:59.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:36:59.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:36:59.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:36:59.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:59.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:59.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:59.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:59.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:59.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:36:59.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:36:59.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:36:59.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:59.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:59.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:59.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:59.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:59.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:36:59.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:36:59.287+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:00.238+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:00.238+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:00.238+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:00.238+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:00.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:00.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:00.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:00.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:00.036+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:00.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:00.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:00.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:00.421+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:01.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:01.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:01.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:01.191+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:01.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:01.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:01.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:01.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:01.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:01.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:01.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:01.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:01.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:01.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:02.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:02.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:02.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:02.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:02.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:02.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:02.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:02.049+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:02.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:02.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:02.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:02.205+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:02.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:02.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:02.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:02.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:02.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:02.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:02.431+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:03.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:03.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:03.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:03.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:03.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:03.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:03.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:03.058+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:03.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:03.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:03.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:03.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:03.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:03.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:03.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:03.247+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:03.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:03.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:03.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:03.390+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:04.242 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:03.957+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32036 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:04.242 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:04.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:04.242 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:04.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:04.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:04.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:04.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:04.070+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:04.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:04.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:04.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:04.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:04.241+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:04.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:04.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:04.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:04.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:04.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:04.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:04.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:04.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:05.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:05.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:05.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:05.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:05.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:05.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:05.235+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:05.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:05.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:05.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:05.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:05.034+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:05.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:05.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:05.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:05.470+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:06.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:06.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:06.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:06.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:06.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:06.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:06.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:06.240+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:06.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:06.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:06.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:06.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:06.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:06.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:06.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:06.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:06.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:06.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:06.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:06.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:07.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:07.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:07.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:07.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:07.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:07.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:07.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:07.208+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:07.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:07.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:07.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:07.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:07.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:07.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:07.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:07.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:07.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:07.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:07.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:07.535+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:08.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:08.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:08.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:08.063+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:08.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:08.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:08.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:08.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:08.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:08.226+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:08.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:08.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:08.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:08.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:08.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:08.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:08.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:08.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:09.237 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:08.963+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32126 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:09.237 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:09.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:09.237 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:09.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:09.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:09.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:09.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:09.079+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:09.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:09.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:09.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:09.236+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:09.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:09.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:09.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:09.526+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:10.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:10.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:10.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:10.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:10.198+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:10.198+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:10.198+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:10.198+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:11.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:10.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:11.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:10.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:11.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:10.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:11.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:10.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:11.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:11.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:11.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:11.227+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:11.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:11.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:11.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:11.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:11.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:11.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:11.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:11.147+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:11.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:11.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:11.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:11.562+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:12.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:12.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:12.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:12.267+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:12.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:12.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:12.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:12.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:13.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:12.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:12.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:12.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:12.601+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:13.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:13.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:13.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:13.277+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:13.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:13.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:13.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:13.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:13.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:13.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:13.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:13.167+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:13.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:13.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:13.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:13.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:13.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:13.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:13.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:13.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:14.231 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:13.968+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32215 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:14.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:14.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:14.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:14.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:14.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:14.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:14.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:14.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:14.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:14.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:14.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:14.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:14.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:14.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:14.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:14.230+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:14.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:14.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:14.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:14.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:15.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:15.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:15.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:15.225+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:15.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:15.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:15.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:15.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:15.171+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:15.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:15.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:15.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:15.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:16.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:16.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:16.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:16.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:16.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:16.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:16.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:16.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:16.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:16.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:16.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:16.241+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:16.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:16.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:16.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:16.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:17.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:17.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:17.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:17.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:17.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:17.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:17.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:17.261+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:17.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:17.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:17.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:17.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:17.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:17.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:17.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:17.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:17.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:17.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:17.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:17.529+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:18.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:18.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:18.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:18.059+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:18.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:18.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:18.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:18.286+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:18.976 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:18.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:18.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:18.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:18.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:18.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:18.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:18.560+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:19.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:18.974+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32305 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:19.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:19.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:19.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:19.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:19.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:19.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:19.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:19.058+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:19.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:19.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:19.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:19.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:19.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:19.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:19.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:19.277+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:19.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:19.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:19.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:19.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:19.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:19.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:19.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:19.516+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:20.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:20.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:20.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:20.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:20.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:20.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:20.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:20.069+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:20.339 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:20.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:20.339 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:20.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:20.339 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:20.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:20.339 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:20.268+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:20.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:20.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:20.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:20.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:21.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:21.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:21.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:21.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:21.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:21.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:21.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:21.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:21.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:21.258+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:21.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:21.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:21.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:21.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:22.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:22.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:22.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:22.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:22.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:22.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:22.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:22.159+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:22.532 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:22.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:22.532 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:22.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:22.533 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:22.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:22.533 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:22.300+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:22.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:22.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:22.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:22.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:22.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:22.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:22.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:22.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:23.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:23.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:23.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:23.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:23.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:23.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:23.520 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:23.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:23.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:23.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:23.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:23.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:23.314+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:23.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:23.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:23.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:23.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:23.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:24.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:23.980+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32393 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:24.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:24.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:24.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:24.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:24.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:24.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:24.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:24.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:24.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:24.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:24.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:24.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:24.472+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:25.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:25.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:25.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:25.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:25.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:25.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:25.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:25.363+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:25.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:25.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:25.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:25.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:25.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:25.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:25.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:25.149+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:25.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:25.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:25.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:25.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:25.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:26.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:26.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:26.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:26.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:26.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:26.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:26.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:26.332+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:26.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:26.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:26.460 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:26.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:26.460 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:26.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:26.460 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:26.105+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:26.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:26.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:26.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:26.457+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:27.380 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:27.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:27.381 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:27.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:27.381 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:27.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:27.381 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:27.118+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:27.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:27.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:27.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:27.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:27.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:27.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:27.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:27.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:27.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:27.418+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:28.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:28.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:28.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:28.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:28.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:28.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:28.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:28.356+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:28.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:28.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:28.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:28.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:28.460 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:28.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:28.460 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:28.112+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:28.457+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:29.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:28.985+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32477 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:29.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:29.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:29.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:29.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:29.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:29.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:29.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:29.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:29.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:29.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:29.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:29.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:29.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:29.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:30.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:30.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:30.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:30.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:30.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:30.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:30.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:30.042+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:30.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:30.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:30.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:30.328+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:30.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:30.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:30.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:30.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:31.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:31.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:31.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:31.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:31.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:31.315+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:31.315+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:31.315+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:31.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:31.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:31.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:31.445+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:32.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:32.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:32.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:32.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:32.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:32.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:32.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:32.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:32.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:32.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:32.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:32.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:32.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:32.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:32.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:32.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:32.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:32.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:32.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:32.435+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:33.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:33.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:33.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:33.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:33.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:33.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:33.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:33.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:33.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:33.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:33.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:33.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:33.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:33.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:33.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:33.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:33.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:33.376+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:34.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:33.991+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32555 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:34.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:34.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:34.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:34.087+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:34.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:34.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:34.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:34.360+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:34.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:34.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:34.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:34.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:34.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:34.498+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:35.388 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:35.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:35.388 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:35.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:35.388 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:35.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:35.388 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:35.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:35.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:35.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:35.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:35.386+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:35.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:35.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:35.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:35.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:35.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:35.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:35.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:35.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:36.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:36.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:36.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:36.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:36.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:36.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:36.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:36.078+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:36.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:36.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:36.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:36.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:36.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:36.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:36.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:36.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:36.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:37.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:37.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:37.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:37.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:37.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:37.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:37.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:37.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:37.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:37.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:37.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:37.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:37.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:37.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:37.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:37.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:37.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:37.399+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:38.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:38.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:38.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:38.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:38.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:38.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:38.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:38.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:38.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:38.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:38.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:38.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:38.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:38.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:38.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:38.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:38.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:39.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:38.997+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32636 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:39.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:39.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:39.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:39.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:39.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:39.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:39.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:39.412+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:39.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:39.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:39.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:39.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:39.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:39.531+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:40.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:40.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:40.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:40.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:40.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:40.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:40.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:40.099+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:40.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:40.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:40.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:40.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:40.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:40.384+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:40.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:40.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:40.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:40.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:41.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:41.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:41.360 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:41.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:41.360 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:41.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:41.360 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:41.052+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:41.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:41.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:41.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:41.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:41.359+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:41.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:41.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:41.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:41.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:41.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:41.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:41.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:41.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:42.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:42.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:42.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:42.072+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:42.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:42.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:42.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:42.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:42.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:42.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:42.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:42.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:42.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:43.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:43.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:43.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:43.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:43.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:43.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:43.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:43.106+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:43.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:43.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:43.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:43.424+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:43.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:43.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:43.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:43.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:43.421+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:44.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:44.003+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32716 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:44.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:44.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:44.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:44.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:44.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:44.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:44.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:44.406+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:44.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:44.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:44.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:44.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:45.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:45.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:45.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:45.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:45.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:45.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:45.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:45.154+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:45.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:45.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:45.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:45.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:45.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:45.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:45.418+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:45.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:45.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:45.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:45.409+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:45.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:45.409+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:45.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:45.409+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:46.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:46.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:46.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:46.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:46.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:46.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:46.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:46.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:46.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:46.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:46.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:46.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:46.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:46.419+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:46.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:46.419+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:46.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:46.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:46.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:46.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:47.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:47.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:47.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:47.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:47.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:47.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:47.445 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:47.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:47.445 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:47.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:47.445 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:47.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:47.445 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:47.415+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:47.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:47.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:47.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:47.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:47.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:47.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:47.443+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:48.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:48.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:48.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:48.231+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:48.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:48.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:48.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:48.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:48.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:48.443+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:48.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:48.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:48.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:48.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:49.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:49.009+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32800 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:49.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:49.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:49.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:49.214+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:49.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:49.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:49.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:49.398+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:49.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:49.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:49.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:49.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:50.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:50.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:50.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:50.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:50.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:50.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:50.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:50.397+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:50.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:50.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:50.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:50.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:50.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:50.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:50.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:50.192+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:50.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:50.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:50.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:50.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:50.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:50.443+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:51.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:51.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:51.438 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:51.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:51.438 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:51.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:51.438 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:51.166+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:51.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:51.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:51.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:51.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:51.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:51.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:51.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:51.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:52.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:52.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:52.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:52.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:52.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:52.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:52.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:52.168+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:52.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:52.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:52.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:52.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:52.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:52.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:52.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:52.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:52.424+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:53.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:53.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:53.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:53.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:53.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:53.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:53.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:53.390+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:53.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:53.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:53.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:53.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:53.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:53.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:53.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:53.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:53.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:53.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:53.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:53.469+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:54.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:54.014+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32886 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:54.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:54.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:54.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:54.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:54.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:54.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:54.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:54.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:54.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:54.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:54.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:54.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:55.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:55.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:55.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:55.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:55.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:55.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:55.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:55.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:55.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:55.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:55.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:55.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:55.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:55.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:55.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:55.447+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:56.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:56.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:56.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:56.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:56.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:56.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:56.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:56.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:56.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:56.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:56.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:56.501+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:56.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:56.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:56.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:56.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:57.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:57.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:57.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:57.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:57.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:57.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:57.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:57.149+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:57.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:57.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:57.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:57.461+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:57.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:57.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:57.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:57.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:58.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:58.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:58.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:58.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:58.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:58.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:58.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:58.105+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:58.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:58.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:58.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:58.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:58.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:58.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:58.796 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:58.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:58.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:58.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:58.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:58.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:58.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:58.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:58.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:58.428+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:59.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:37:59.020+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 32976 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:37:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:59.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:59.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:59.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:37:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:37:59.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:37:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:59.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:59.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:59.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:37:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:37:59.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:37:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:59.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:59.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:59.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:37:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:37:59.382+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:00.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:00.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:00.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:00.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:00.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:00.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:00.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:00.337+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:00.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:00.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:00.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:00.144+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:00.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:00.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:00.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:00.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:00.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:00.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:00.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:00.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:01.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:01.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:01.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:01.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:01.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:01.289+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:01.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:01.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:01.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:01.168+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:01.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:01.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:01.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:01.336+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:02.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:02.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:02.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:02.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:02.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:02.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:02.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:02.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:02.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:02.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:02.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:02.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:03.392 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:03.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:03.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:03.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:03.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:03.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:03.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:03.363+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:03.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:03.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:03.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:03.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:03.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:03.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:03.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:03.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:03.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:03.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:03.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:03.391+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:04.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:04.026+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33064 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:04.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:04.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:04.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:04.082+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:04.388+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:04.388+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:04.388+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:04.389+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:04.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:04.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:04.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:04.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:05.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:05.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:05.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:05.035+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:05.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:05.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:05.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:05.388+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:05.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:05.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:05.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:05.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:06.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:06.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:06.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:06.028+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:06.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:06.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:06.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:06.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:06.361+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:06.361+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:06.361+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:06.362+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:06.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:06.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:06.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:06.980+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:07.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:07.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:07.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:07.343+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:07.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:07.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:07.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:07.363+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:08.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:08.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:08.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:08.023+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:08.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:08.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:08.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:08.301+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:08.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:08.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:08.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:08.385+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:09.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:09.031+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33154 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:09.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:09.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:09.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:09.069+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:09.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:09.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:09.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:09.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:09.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:09.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:09.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:09.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:10.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:10.272+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:10.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:10.272+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:10.272+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:10.273+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:10.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:10.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:10.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:10.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:10.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:10.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:10.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:10.033+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:10.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:10.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:10.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:10.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:10.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:10.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:11.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:10.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:11.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:10.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:11.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:10.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:11.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:10.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:11.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:11.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:11.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:11.242+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:11.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:11.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:11.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:11.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:12.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:11.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:12.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:11.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:12.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:11.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:12.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:12.000+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:12.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:12.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:12.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:12.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:12.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:12.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:12.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:12.283+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:13.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:13.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:13.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:13.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:13.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:13.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:13.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:13.302+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:13.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:13.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:13.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:13.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:14.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:14.037+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33243 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:14.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:14.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:14.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:14.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:14.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:14.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:14.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:14.078+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:14.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:14.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:14.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:14.288+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:14.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:14.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:14.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:14.367+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:15.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:15.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:15.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:15.327+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:15.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:15.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:15.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:15.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:15.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:15.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:15.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:15.298+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:15.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:15.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:15.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:15.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:15.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:15.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:15.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:15.125+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:16.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:16.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:16.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:16.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:16.316+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:16.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:16.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:16.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:16.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:16.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:16.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:16.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:16.267+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:16.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:16.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:16.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:16.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:16.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:16.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:16.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:16.124+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:17.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:17.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:17.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:17.339+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:17.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:17.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:17.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:17.163+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:17.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:17.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:17.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:17.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:17.294+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:18.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:18.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:18.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:18.262+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:18.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:18.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:18.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:18.189+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:18.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:18.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:18.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:18.333+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:19.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:19.042+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33333 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:19.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:19.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:19.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:19.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:19.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:19.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:19.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:19.230+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:19.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:19.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:19.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:19.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:19.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:19.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:19.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:19.158+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:19.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:19.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:19.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:19.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:19.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:19.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:19.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:19.288+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:20.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:20.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:20.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:20.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:20.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:20.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:20.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:20.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:20.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:20.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:20.169+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:20.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:20.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:20.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:20.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:20.234+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:21.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:21.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:21.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:21.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:21.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:21.287+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:21.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:21.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:21.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:21.242+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:21.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:21.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:21.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:21.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:22.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:22.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:22.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:22.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:22.248+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:22.248+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:22.248+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:22.249+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:22.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:22.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:22.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:22.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:22.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:22.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:22.191+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:23.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:23.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:23.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:23.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:23.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:23.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:23.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:23.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:23.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:23.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:23.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:23.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:23.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:23.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:23.274+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:23.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:23.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:23.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:23.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:23.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:23.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:23.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:23.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:24.315 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:24.048+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33421 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:24.316 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:24.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:24.316 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:24.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:24.316 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:24.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:24.316 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:24.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:24.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:24.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:24.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:24.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:24.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:24.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:24.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:24.298+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:24.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:24.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:24.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:24.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:24.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:24.314+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:25.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:25.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:25.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:25.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:25.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:25.316+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:25.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:25.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:25.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:25.270+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:25.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:25.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:25.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:25.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:25.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:25.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:25.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:25.230+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:26.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:26.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:26.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:26.300+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:26.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:26.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:26.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:26.223+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:26.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:26.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:26.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:26.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:26.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:26.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:26.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:26.327+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:27.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:27.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:27.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:27.350+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:27.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:27.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:27.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:27.184+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:27.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:27.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:27.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:27.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:28.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:28.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:28.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:28.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:28.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:28.320+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:28.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:28.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:28.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:28.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:28.208+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:28.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:28.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:28.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:28.329+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:29.329 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:29.054+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33511 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:29.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:29.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:29.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:29.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:29.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:29.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:29.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:29.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:29.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:29.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:29.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:29.329+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:29.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:29.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:29.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:29.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:29.371+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:30.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:30.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:30.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:30.354+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:30.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:30.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:30.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:30.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:30.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:30.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:30.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:30.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:30.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:30.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:30.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:30.340+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:31.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:31.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:31.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:31.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:31.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:31.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:31.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:31.360+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:31.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:31.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:31.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:31.299+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:32.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:32.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:32.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:32.314+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:32.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:32.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:32.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:32.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:32.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:32.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:32.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:32.263+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:32.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:32.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:32.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:32.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:32.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:32.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:32.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:32.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:33.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:33.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:33.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:33.361+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:33.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:33.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:33.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:33.232+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:33.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:33.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:33.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:33.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:34.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:34.060+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33600 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:34.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:34.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:34.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:34.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:34.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:34.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:34.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:34.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:34.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:34.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:34.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:34.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:34.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:34.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:34.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:34.259+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:34.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:34.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:34.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:34.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:35.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:35.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:35.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:35.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:35.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:35.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:35.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:35.238+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:35.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:35.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:35.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:35.422+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:36.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:36.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:36.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:36.435+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:36.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:36.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:36.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:36.263+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:36.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:36.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:36.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:36.194+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:37.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:37.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:37.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:37.204+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:37.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:37.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:37.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:37.283+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:37.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:37.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:37.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:37.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:37.445+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:38.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:38.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:38.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:38.232+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:38.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:38.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:38.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:38.298+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:38.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:38.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:38.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:38.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:39.491 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:39.067+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33690 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:39.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:39.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:39.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:39.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:39.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:39.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:39.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:39.251+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:39.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:39.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:39.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:39.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:39.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:39.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:39.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:39.222+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:39.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:39.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:39.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:39.492+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:40.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:40.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:40.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:40.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:40.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:40.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:40.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:40.297+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:40.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:40.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:40.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:40.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:40.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:40.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:40.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:40.198+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:40.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:40.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:40.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:40.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:40.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:40.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:40.506+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:41.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:41.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:41.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:41.243+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:41.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:41.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:41.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:41.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:41.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:41.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:41.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:41.323+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:41.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:41.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:41.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:41.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:41.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:41.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:41.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:41.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:42.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:42.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:42.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:42.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:42.339+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:42.339+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:42.339+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:42.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:42.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:42.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:42.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:42.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:43.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:43.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:43.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:43.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:43.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:43.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:43.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:43.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:43.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:43.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:43.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:43.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:43.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:43.499+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:43.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:43.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:43.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:43.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:43.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:43.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:43.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:43.327+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:44.368 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:44.072+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33777 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:44.368 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:44.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:44.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:44.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:44.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:44.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:44.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:44.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:44.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:44.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:44.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:44.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:44.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:44.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:44.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:44.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:45.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:45.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:45.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:45.372+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:45.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:45.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:45.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:45.466+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:45.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:45.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:45.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:45.304+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:46.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:46.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:46.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:46.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:46.389+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:46.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:46.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:46.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:46.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:46.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:46.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:46.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:46.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:46.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:46.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:46.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:46.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:46.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:46.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:47.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:47.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:47.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:47.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:47.416+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:47.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:47.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:47.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:47.456+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:47.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:47.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:47.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:47.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:47.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:47.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:47.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:47.254+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:48.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:48.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:48.459+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:48.460+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:48.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:48.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:48.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:48.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:48.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:48.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:48.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:48.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:48.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:48.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:48.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:48.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:48.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:48.240+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:49.381 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:49.078+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33867 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:49.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:49.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:49.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:49.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:49.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:49.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:49.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:49.201+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:49.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:49.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:49.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:49.380+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:49.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:49.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:49.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:49.496+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:50.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:50.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:50.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:50.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:50.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:50.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:50.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:50.372+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:50.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:50.175+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:50.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:50.175+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:50.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:50.175+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:50.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:50.175+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:50.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:50.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:50.451+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:50.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:50.452+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:51.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:51.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:51.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:51.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:51.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:51.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:51.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:51.340+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:51.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:51.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:51.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:51.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:51.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:51.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:51.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:51.138+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:51.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:51.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:51.500+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:51.501+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:52.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:52.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:52.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:52.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:52.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:52.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:52.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:52.152+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:52.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:52.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:52.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:52.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:52.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:52.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:52.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:52.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:52.527+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:53.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:53.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:53.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:53.154+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:53.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:53.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:53.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:53.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:53.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:53.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:53.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:53.542+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:54.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:54.084+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 33955 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:54.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:54.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:54.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:54.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:54.196+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:54.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:54.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:54.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:54.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:54.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:54.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:54.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:54.590+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:55.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:55.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:55.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:55.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:55.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:55.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:55.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:55.215+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:55.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:55.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:55.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:55.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:55.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:55.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:55.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:55.270+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:55.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:55.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:55.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:56.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:55.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:56.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:56.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:56.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:56.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:56.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:56.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:56.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:56.192+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:56.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:56.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:56.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:56.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:56.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:56.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:56.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:56.286+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:56.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:56.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:56.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:56.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:56.522+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:57.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:57.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:57.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:57.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:57.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:57.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:57.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:57.177+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:57.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:57.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:57.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:57.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:57.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:57.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:57.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:57.320+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:57.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:57.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:57.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:57.487+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:58.460 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-02-19T19:38:58.465 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-19T19:38:58.466 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-02-19T19:38:58.467 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-19T19:38:58.479 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-02-19T19:38:58.490 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task sleep... 2024-02-19T19:38:58.491 INFO:teuthology.task.sleep:Sleeping for 3600 seconds 2024-02-19T19:38:58.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:58.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:58.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:58.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:58.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:58.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:58.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:58.313+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:58.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:58.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:58.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:58.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:58.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:58.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:58.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:58.181+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:58.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:58.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:58.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:58.523+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:38:59.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:38:59.090+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34045 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:38:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:59.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:59.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:59.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:38:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:38:59.228+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:59.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:59.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:59.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:59.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:59.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:59.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:38:59.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:38:59.331+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:38:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:59.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:59.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:59.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:38:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:38:59.556+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:00.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:00.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:00.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:00.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:00.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:00.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:00.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:00.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:00.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:00.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:00.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:00.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:00.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:00.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:00.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:00.243+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:00.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:00.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:00.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:00.514+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:01.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:01.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:01.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:01.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:01.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:01.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:01.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:01.263+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:01.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:01.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:01.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:01.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:01.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:01.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:01.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:01.261+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:01.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:01.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:01.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:01.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:01.515+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:02.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:02.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:02.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:02.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:02.287+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:02.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:02.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:02.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:02.305+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:02.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:02.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:02.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:02.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:02.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:02.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:02.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:02.512+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:03.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:03.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:03.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:03.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:03.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:03.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:03.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:03.330+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:03.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:03.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:03.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:03.479+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:04.371 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:04.096+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34132 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:04.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:04.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:04.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:04.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:04.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:04.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:04.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:04.316+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:04.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:04.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:04.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:04.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:04.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:04.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:04.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:04.450+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:05.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:05.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:05.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:05.365+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:05.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:05.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:05.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:05.341+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:05.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:05.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:05.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:05.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:05.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:05.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:05.440+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:06.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:06.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:06.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:06.407+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:06.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:06.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:06.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:06.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:06.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:06.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:06.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:06.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:06.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:06.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:06.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:06.436+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:07.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:07.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:07.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:07.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:07.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:07.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:07.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:07.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:07.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:07.256+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:07.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:07.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:07.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:07.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:07.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:07.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:07.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:07.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:08.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:08.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:08.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:08.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:08.284+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:08.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:08.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:08.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:08.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:08.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:08.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:08.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:08.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:08.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:08.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:08.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:08.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:08.420+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:09.378 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:09.102+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34221 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:09.378 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:09.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:09.378 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:09.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:09.379 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:09.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:09.379 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:09.247+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:09.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:09.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:09.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:09.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:09.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:09.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:09.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:09.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:10.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:10.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:10.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:10.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:10.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:10.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:10.262+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:10.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:10.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:10.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:10.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:10.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:10.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:10.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:10.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:10.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:10.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:10.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:10.348+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:11.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:11.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:11.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:11.275+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:11.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:11.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:11.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:11.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:11.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:11.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:11.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:11.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:11.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:11.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:11.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:11.312+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:12.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:12.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:12.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:12.276+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:12.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:12.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:12.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:12.322+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:12.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:12.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:12.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:12.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:12.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:12.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:12.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:12.313+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:13.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:13.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:13.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:13.291+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:13.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:13.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:13.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:13.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:13.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:13.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:13.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:13.303+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:13.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:13.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:13.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:13.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:13.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:13.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:13.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:13.281+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:14.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:14.109+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34310 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:14.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:14.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:14.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:14.312+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:14.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:14.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:14.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:14.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:14.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:14.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:14.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:14.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:14.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:14.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:15.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:15.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:15.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:15.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:15.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:15.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:15.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:15.315+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:15.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:15.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:15.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:15.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:16.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:16.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:16.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:16.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:16.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:16.303+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:16.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:16.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:16.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:16.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:16.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:16.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:16.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:16.351+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:16.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:16.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:16.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:16.393+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:17.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:17.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:17.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:17.269+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:17.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:17.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:17.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:17.382+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:17.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:17.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:17.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:17.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:17.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:17.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:17.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:17.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:18.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:18.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:18.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:18.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:18.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:18.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:18.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:18.385+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:18.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:18.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:18.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:18.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:18.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:19.384 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:19.115+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34400 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:19.384 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:19.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:19.384 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:19.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:19.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:19.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:19.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:19.246+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:19.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:19.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:19.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:19.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:19.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:19.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:19.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:19.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:19.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:19.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:19.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:19.383+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:20.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:20.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:20.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:20.343+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:20.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:20.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:20.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:20.260+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:20.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:20.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:20.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:20.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:20.383+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:21.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:21.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:21.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:21.218+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:21.320+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:21.320+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:21.320+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:21.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:21.321+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:21.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:21.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:21.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:21.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:21.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:21.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:21.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:21.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:22.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:22.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:22.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:22.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:22.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:22.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:22.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:22.258+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:22.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:22.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:22.414 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:22.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:22.414 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:22.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:22.414 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:22.273+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:22.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:22.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:22.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:22.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:22.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:22.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:22.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:22.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:23.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:23.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:23.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:23.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:23.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:23.380+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:23.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:23.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:23.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:23.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:23.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:23.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:23.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:23.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:23.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:23.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:23.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:23.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:23.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:23.261+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:24.417 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:24.121+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34490 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:24.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:24.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:24.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:24.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:24.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:24.232+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:24.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:24.233+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:24.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:24.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:24.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:24.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:24.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:24.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:24.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:24.319+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:24.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:24.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:24.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:24.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:25.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:25.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:25.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:25.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:25.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:25.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:25.214+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:25.215+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:25.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:25.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:25.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:25.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:25.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:25.336+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:26.447 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:26.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:26.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:26.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:26.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:26.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:26.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:26.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:26.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:26.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:26.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:26.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:26.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:26.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:26.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:26.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:26.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:26.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:26.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:26.446+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:27.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:27.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:27.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:27.164+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:27.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:27.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:27.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:27.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:27.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:27.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:27.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:27.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:27.401+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:28.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:28.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:28.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:28.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:28.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:28.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:28.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:28.147+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:28.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:28.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:28.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:28.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:28.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:28.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:28.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:28.364+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:28.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:28.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:28.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:28.401+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:29.389 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:29.127+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34581 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:29.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:29.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:29.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:29.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:29.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:29.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:29.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:29.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:29.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:29.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:29.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:29.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:29.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:29.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:29.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:29.188+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:29.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:29.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:29.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:29.389+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:30.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:30.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:30.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:30.218+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:30.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:30.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:30.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:30.395+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:30.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:30.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:30.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:30.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:30.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:30.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:30.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:30.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:31.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:31.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:31.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:31.237+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:31.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:31.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:31.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:31.416+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:31.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:31.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:31.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:31.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:31.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:31.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:31.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:31.342+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:32.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:32.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:32.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:32.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:32.463+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:32.463+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:32.463+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:32.464+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:32.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:32.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:32.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:32.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:32.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:32.333+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:33.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:33.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:33.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:33.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:33.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:33.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:33.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:33.261+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:33.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:33.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:33.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:33.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:33.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:33.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:33.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:33.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:33.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:33.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:33.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:33.494+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:34.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:34.132+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34668 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:34.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:34.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:34.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:34.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:34.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:34.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:34.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:34.318+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:34.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:34.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:34.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:34.541+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:35.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:35.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:35.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:35.283+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:35.334+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:35.334+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:35.334+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:35.334+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:35.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:35.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:35.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:36.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:35.570+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:36.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:36.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:36.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:36.264+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:36.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:36.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:36.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:36.329+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:36.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:36.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:36.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:36.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:37.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:37.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:37.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:37.268+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:37.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:37.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:37.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:37.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:37.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:37.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:37.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:37.594+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:38.375 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:38.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:38.376 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:38.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:38.376 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:38.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:38.376 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:38.227+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:38.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:38.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:38.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:38.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:38.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:38.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:38.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:38.608+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:38.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:38.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:38.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:38.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:39.394 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:39.139+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34759 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:39.394 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:39.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:39.394 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:39.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:39.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:39.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:39.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:39.217+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:39.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:39.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:39.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:39.393+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:39.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:39.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:39.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:39.628+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:40.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:40.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:40.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:40.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:40.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:40.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:40.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:40.261+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:40.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:40.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:40.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:40.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:40.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:40.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:40.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:40.628+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:41.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:41.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:41.650 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:41.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:41.650 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:41.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:41.650 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:41.301+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:41.650 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:41.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:41.650 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:41.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:41.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:41.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:41.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:41.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:41.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:41.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:41.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:42.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:41.675+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:42.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:42.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:42.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:42.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:42.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:42.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:42.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:42.297+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:42.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:42.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:42.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:42.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:42.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:42.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:42.672 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:42.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:42.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:42.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:42.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:42.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:43.646 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:43.347+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:43.646 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:43.347+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:43.646 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:43.347+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:43.646 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:43.348+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:43.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:43.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:43.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:43.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:43.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:43.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:43.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:43.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:43.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:43.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:43.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:43.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:44.467 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:44.145+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34847 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:44.467 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:44.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:44.467 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:44.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:44.467 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:44.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:44.467 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:44.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:44.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:44.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:44.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:44.616+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:44.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:44.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:44.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:44.466+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:45.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:45.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:45.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:45.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:45.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:45.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:45.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:45.579+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:45.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:45.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:45.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:45.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:46.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:46.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:46.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:46.358+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:46.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:46.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:46.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:46.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:46.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:46.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:46.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:46.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:46.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:46.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:46.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:46.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:46.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:47.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:47.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:47.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:47.405+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:47.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:47.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:47.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:47.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:47.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:47.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:47.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:47.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:47.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:47.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:47.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:47.601+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:48.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:48.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:48.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:48.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:48.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:48.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:48.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:48.557+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:48.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:48.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:48.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:48.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:49.457 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:49.151+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 34938 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:49.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:49.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:49.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:49.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:49.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:49.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:49.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:49.388+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:49.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:49.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:49.527+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:49.528+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:49.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:49.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:49.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:49.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:50.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:50.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:50.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:50.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:50.356+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:50.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:50.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:50.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:50.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:50.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:50.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:50.866 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:50.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:50.866 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:50.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:50.866 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:50.563+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:51.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:51.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:51.535+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:51.536+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:51.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:51.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:51.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:51.355+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:51.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:51.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:51.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:51.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:51.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:51.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:51.478+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:52.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:52.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:52.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:52.513+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:52.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:52.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:52.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:52.378+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:52.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:52.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:52.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:52.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:52.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:52.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:52.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:53.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:53.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:53.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:53.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:53.501+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:53.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:53.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:53.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:53.390+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:53.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:53.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:53.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:53.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:53.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:53.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:53.396+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:54.417 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:54.157+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35028 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:54.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:54.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:54.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:54.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:54.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:54.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:54.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:54.398+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:54.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:54.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:54.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:54.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:54.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:54.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:54.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:54.520+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:55.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:55.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:55.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:55.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:55.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:55.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:55.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:55.487+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:55.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:55.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:55.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:55.422+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:55.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:55.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:55.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:55.356+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:56.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:56.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:56.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:56.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:56.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:56.479+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:56.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:56.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:56.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:56.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:56.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:56.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:56.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:56.396+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:57.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:57.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:57.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:57.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:57.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:57.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:57.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:57.438+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:57.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:57.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:57.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:57.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:57.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:57.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:57.406+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:58.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:58.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:58.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:58.526+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:58.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:58.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:58.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:58.477+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:58.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:58.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:58.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:58.423+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:59.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:39:59.163+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35118 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:39:59.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:59.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:59.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:59.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:59.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:59.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:39:59.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:39:59.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:39:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:59.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:59.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:59.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:39:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:39:59.513+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:39:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:59.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:59.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:59.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:39:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:39:59.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:00.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:00.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:00.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:00.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:00.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:00.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:00.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:00.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:00.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:00.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:00.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:00.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:00.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:00.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:00.520+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:00.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:00.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:00.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:00.538+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:01.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:01.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:01.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:01.536+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:01.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:01.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:01.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:01.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:01.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:01.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:01.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:01.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:01.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:01.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:01.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:01.573+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:02.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:02.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:02.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:02.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:02.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:02.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:02.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:02.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:02.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:02.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:02.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:02.485+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:02.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:02.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:02.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:02.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:02.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:02.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:02.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:02.549+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:03.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:03.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:03.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:03.552+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:03.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:03.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:03.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:03.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:03.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:03.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:03.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:03.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:03.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:03.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:03.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:03.569+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:04.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:04.169+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35208 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:04.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:04.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:04.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:04.572+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:04.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:04.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:04.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:04.512+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:04.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:04.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:04.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:04.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:04.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:04.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:04.526+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:05.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:05.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:05.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:05.536+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:05.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:05.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:05.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:05.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:05.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:05.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:05.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:05.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:05.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:05.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:05.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:06.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:06.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:06.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:06.584+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:06.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:06.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:06.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:06.526+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:06.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:06.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:06.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:06.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:06.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:06.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:06.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:06.554+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:07.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:07.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:07.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:07.590+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:07.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:07.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:07.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:07.572+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:07.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:07.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:07.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:07.519+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:08.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:08.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:08.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:08.565+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:08.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:08.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:08.522+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:08.523+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:08.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:08.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:08.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:08.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:08.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:08.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:08.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:08.547+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:09.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:09.176+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35299 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:09.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:09.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:09.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:09.576+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:09.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:09.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:09.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:09.541+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:09.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:09.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:09.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:09.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:09.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:09.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:09.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:11.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:10.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:11.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:10.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:10.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:10.562+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:10.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:10.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:10.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:10.571+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:11.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:10.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:11.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:10.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:11.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:10.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:11.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:10.558+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:11.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:11.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:11.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:11.594+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:11.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:11.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:11.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:11.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:11.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:11.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:11.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:11.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:11.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:11.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:11.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:11.565+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:12.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:12.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:12.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:12.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:12.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:12.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:12.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:12.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:12.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:12.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:12.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:12.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:12.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:12.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:12.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:12.604+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:12.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:12.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:12.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:12.624+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:14.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:13.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:14.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:13.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:14.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:13.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:14.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:13.639+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:13.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:13.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:13.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:13.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:13.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:14.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:13.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:14.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:13.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:14.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:13.580+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:14.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:14.182+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35389 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:14.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:14.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:14.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:14.563+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:14.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:14.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:14.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:14.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:14.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:15.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:14.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:15.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:14.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:15.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:14.605+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:15.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:15.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:15.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:15.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:15.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:15.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:15.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:15.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:16.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:15.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:16.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:15.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:16.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:15.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:16.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:15.566+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:16.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:16.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:16.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:16.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:16.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:16.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:16.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:16.597+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:16.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:16.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:16.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:16.504+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:16.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:16.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:16.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:16.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:16.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:17.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:17.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:17.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:17.631+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:17.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:17.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:17.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:17.531+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:17.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:17.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:17.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:17.547+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:18.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:18.619+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:18.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:18.619+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:18.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:18.619+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:18.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:18.620+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:18.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:18.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:18.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:18.567+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:18.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:18.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:18.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:18.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:18.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:18.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:18.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:18.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:19.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:19.188+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35479 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:19.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:19.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:19.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:19.626+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:19.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:19.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:19.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:19.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:19.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:19.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:19.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:19.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:19.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:19.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:19.528+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:20.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:20.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:20.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:20.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:20.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:20.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:20.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:20.632+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:20.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:20.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:20.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:20.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:20.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:20.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:20.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:20.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:20.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:20.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:20.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:20.501+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:21.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:21.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:21.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:21.510+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:21.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:21.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:21.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:21.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:21.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:21.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:21.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:21.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:21.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:21.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:21.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:21.636+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:22.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:22.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:22.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:22.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:22.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:22.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:22.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:22.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:22.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:23.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:22.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:23.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:22.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:23.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:22.614+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:23.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:23.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:23.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:23.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:23.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:23.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:23.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:23.659+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:23.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:23.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:23.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:23.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:23.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:23.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:23.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:23.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:23.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:23.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:23.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:23.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:24.471 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:24.194+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35569 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:24.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:24.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:24.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:24.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:24.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:24.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:24.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:24.688+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:24.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:24.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:24.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:24.471+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:24.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:24.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:24.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:24.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:25.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:25.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:25.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:25.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:25.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:25.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:25.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:25.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:25.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:25.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:25.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:25.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:25.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:25.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:25.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:25.578+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:26.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:26.684+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:26.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:26.684+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:26.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:26.684+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:26.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:26.684+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:26.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:26.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:26.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:26.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:26.512+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:26.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:26.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:26.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:26.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:26.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:26.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:26.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:26.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:27.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:27.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:27.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:27.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:27.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:27.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:27.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:27.720+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:27.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:27.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:27.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:27.551+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:27.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:27.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:27.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:27.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:28.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:28.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:28.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:28.728+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:28.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:28.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:28.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:28.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:28.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:28.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:28.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:28.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:28.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:28.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:28.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:28.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:28.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:28.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:29.476 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:29.201+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35660 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:29.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:29.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:29.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:29.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:29.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:29.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:29.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:29.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:29.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:29.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:29.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:29.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:29.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:29.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:29.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:29.475+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:30.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:29.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:30.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:29.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:30.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:29.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:30.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:29.755+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:30.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:30.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:30.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:30.557+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:30.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:30.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:30.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:30.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:30.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:30.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:30.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:30.797+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:31.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:31.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:31.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:31.542+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:31.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:31.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:31.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:31.597+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:32.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:31.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:32.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:31.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:32.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:31.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:32.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:31.754+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:32.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:32.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:32.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:32.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:32.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:32.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:32.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:32.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:32.802+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:33.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:33.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:33.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:33.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:33.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:33.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:33.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:33.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:33.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:33.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:33.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:33.639+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:33.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:33.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:33.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:33.791+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:34.492 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:34.207+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35750 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:34.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:34.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:34.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:34.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:34.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:34.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:34.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:34.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:34.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:34.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:34.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:34.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:34.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:34.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:34.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:34.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:34.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:34.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:34.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:34.769+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:35.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:35.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:35.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:35.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:35.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:35.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:35.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:35.600+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:35.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:35.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:35.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:35.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:35.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:35.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:35.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:35.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:35.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:35.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:35.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:35.783+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:36.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:36.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:36.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:36.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:36.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:36.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:36.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:36.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:36.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:36.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:36.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:36.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:36.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:36.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:36.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:36.496+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:36.782+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:37.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:37.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:37.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:37.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:37.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:37.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:37.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:37.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:37.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:37.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:37.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:37.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:37.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:37.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:38.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:37.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:38.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:38.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:38.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:38.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:38.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:38.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:38.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:38.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:38.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:38.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:38.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:38.678+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:38.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:38.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:38.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:38.785+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:39.488 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:39.213+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35840 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:39.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:39.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:39.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:39.688+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:39.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:39.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:39.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:39.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:40.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:39.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:40.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:39.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:40.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:39.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:40.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:39.824+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:40.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:40.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:40.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:40.442+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:40.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:40.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:40.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:40.694+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:40.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:40.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:40.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:40.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:41.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:41.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:41.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:41.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:41.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:41.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:41.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:41.670+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:41.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:41.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:41.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:41.879+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:42.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:42.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:42.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:42.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:42.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:42.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:42.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:42.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:42.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:42.679+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:43.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:42.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:43.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:42.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:43.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:42.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:43.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:42.878+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:43.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:43.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:43.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:43.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:43.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:43.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:43.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:43.501+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:44.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:43.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:44.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:43.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:44.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:43.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:44.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:43.901+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:44.479 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:44.220+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 35930 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:44.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:44.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:44.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:44.691+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:44.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:44.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:44.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:44.479+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:44.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:44.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:44.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:44.883+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:45.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:45.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:45.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:45.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:45.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:45.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:45.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:45.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:45.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:45.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:45.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:45.903+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:46.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:46.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:46.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:46.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:46.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:46.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:46.513+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:46.514+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:46.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:46.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:46.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:46.924+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:47.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:47.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:47.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:47.703+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:47.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:47.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:47.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:47.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:47.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:47.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:47.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:47.913+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:48.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:48.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:48.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:48.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:48.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:48.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:48.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:48.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:49.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:48.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:49.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:48.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:49.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:48.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:49.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:48.918+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:49.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:49.226+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36021 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:49.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:49.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:49.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:49.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:49.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:49.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:49.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:49.525+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:50.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:49.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:50.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:49.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:49.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:49.751+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:50.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:49.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:50.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:49.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:50.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:49.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:50.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:49.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:50.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:50.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:50.882 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:50.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:50.882 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:50.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:50.882 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:50.767+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:50.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:50.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:50.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:50.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:50.883 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:50.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:50.883 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:50.572+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:50.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:50.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:50.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:50.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:51.833 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:51.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:51.833 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:51.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:51.833 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:51.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:51.834 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:51.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:51.834 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:51.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:51.834 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:51.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:51.834 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:51.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:51.834 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:51.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:51.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:51.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:51.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:51.833+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:52.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:52.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:52.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:52.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:52.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:52.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:52.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:52.572+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:52.862+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:52.862+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:52.862+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:52.863+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:53.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:53.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:53.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:53.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:53.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:53.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:53.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:53.680+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:53.862 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:53.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:53.862 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:53.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:53.862 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:53.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:53.862 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:53.592+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:54.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:53.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:54.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:53.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:54.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:53.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:54.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:53.860+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:54.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:54.232+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36111 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:54.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:54.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:54.912 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:54.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:54.912 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:54.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:54.912 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:54.663+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:54.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:54.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:54.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:54.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:54.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:54.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:54.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:54.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:54.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:54.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:54.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:54.911+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:55.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:55.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:55.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:55.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:55.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:55.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:55.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:55.662+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:55.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:55.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:55.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:55.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:55.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:55.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:55.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:55.605+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:55.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:55.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:55.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:55.869+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:56.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:56.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:56.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:56.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:56.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:56.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:56.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:56.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:56.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:56.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:56.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:56.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:56.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:56.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:56.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:56.604+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:56.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:56.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:56.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:56.913+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:57.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:57.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:57.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:57.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:57.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:57.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:57.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:57.627+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:57.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:57.595+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:57.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:57.595+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:57.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:57.595+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:57.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:57.595+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:57.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:57.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:57.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:57.940+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:58.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:58.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:58.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:58.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:58.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:58.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:58.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:58.639+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:58.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:58.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:58.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:58.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:58.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:58.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:58.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:58.576+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:59.240 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:58.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:59.240 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:58.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:59.240 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:58.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:40:59.240 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:58.985+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:40:59.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:40:59.238+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36201 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:40:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:59.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:59.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:59.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:40:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:40:59.629+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:40:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:59.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:59.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:59.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:40:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:40:59.607+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:00.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:59.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:00.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:59.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:00.240 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:59.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:00.240 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:40:59.986+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:01.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:00.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:01.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:00.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:01.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:00.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:01.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:00.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:01.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:00.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:01.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:00.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:01.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:00.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:01.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:00.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:01.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:01.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:01.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:01.035+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:02.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:01.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:02.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:01.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:02.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:01.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:02.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:01.620+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:02.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:01.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:02.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:01.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:02.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:01.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:02.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:01.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:02.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:02.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:02.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:02.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:03.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:02.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:03.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:02.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:03.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:02.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:03.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:02.552+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:02.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:02.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:02.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:02.620+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:03.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:03.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:03.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:03.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:03.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:03.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:03.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:03.064+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:03.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:03.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:03.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:03.654+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:03.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:03.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:03.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:03.586+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:04.371 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:04.244+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36291 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:04.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:04.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:04.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:04.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:04.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:04.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:04.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:04.050+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:04.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:04.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:04.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:04.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:04.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:04.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:04.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:04.607+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:05.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:04.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:05.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:04.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:05.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:04.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:05.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:04.667+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:05.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:05.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:05.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:05.025+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:05.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:05.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:05.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:05.639+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:06.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:05.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:05.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:05.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:05.620+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:06.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:06.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:06.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:06.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:06.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:06.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:06.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:06.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:06.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:06.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:06.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:06.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:07.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:07.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:07.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:07.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:07.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:07.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:07.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:07.588+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:07.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:07.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:07.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:07.683+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:08.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:08.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:08.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:08.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:09.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:08.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:09.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:08.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:09.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:08.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:09.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:08.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:09.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:08.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:09.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:08.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:09.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:08.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:09.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:08.586+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:09.376 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:09.250+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36382 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:09.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:09.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:09.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:09.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:09.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:09.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:09.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:09.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:09.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:09.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:09.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:09.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:09.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:09.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:09.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:09.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:09.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:09.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:09.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:09.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:10.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:10.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:10.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:10.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:10.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:10.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:10.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:10.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:10.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:10.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:10.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:10.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:10.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:10.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:10.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:10.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:10.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:10.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:10.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:10.657+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:11.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:11.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:11.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:11.146+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:11.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:11.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:11.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:11.749+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:11.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:12.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:11.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:12.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:11.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:12.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:11.683+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:12.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:12.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:12.118+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:12.119+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:13.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:12.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:13.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:12.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:13.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:12.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:13.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:12.751+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:12.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:12.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:12.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:12.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:13.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:13.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:13.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:13.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:14.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:13.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:14.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:13.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:14.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:13.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:14.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:13.713+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:13.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:13.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:13.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:13.616+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:14.383 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:14.257+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36472 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:14.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:14.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:14.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:14.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:14.384 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:14.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:14.384 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:14.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:14.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:14.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:14.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:14.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:14.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:14.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:14.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:14.582+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:14.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:14.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:14.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:14.731+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:15.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:15.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:15.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:15.182+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:15.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:15.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:15.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:15.745+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:15.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:15.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:15.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:15.562+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:16.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:16.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:16.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:16.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:16.146+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:16.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:16.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:16.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:16.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:16.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:16.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:16.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:16.705+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:17.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:17.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:17.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:17.170+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:17.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:18.067 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:17.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:18.068 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:17.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:18.068 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:17.635+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:18.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:17.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:18.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:17.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:18.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:17.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:18.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:17.728+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:18.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:18.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:18.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:18.170+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:18.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:18.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:18.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:18.655+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:18.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:18.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:18.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:18.754+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:19.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:19.263+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36562 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:19.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:19.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:19.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:19.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:19.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:19.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:19.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:19.696+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:19.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:19.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:19.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:19.733+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:20.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:20.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:20.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:20.185+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:21.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:20.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:21.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:20.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:21.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:20.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:21.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:20.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:20.694+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:20.694+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:20.694+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:20.694+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:21.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:21.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:21.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:21.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:21.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:21.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:21.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:21.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:21.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:21.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:21.738+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:21.739+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:22.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:22.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:22.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:22.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:22.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:22.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:22.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:22.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:22.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:22.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:22.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:22.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:22.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:22.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:22.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:22.735+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:23.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:23.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:23.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:23.128+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:23.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:23.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:23.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:23.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:23.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:23.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:23.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:24.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:23.765+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:24.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:24.270+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36652 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:24.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:24.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:24.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:24.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:24.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:24.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:24.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:24.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:25.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:24.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:25.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:24.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:25.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:24.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:25.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:24.754+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:24.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:24.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:24.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:24.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:25.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:25.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:25.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:25.149+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:25.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:25.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:25.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:25.715+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:26.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:25.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:25.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:25.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:25.712+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:26.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:26.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:26.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:26.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:26.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:26.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:26.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:26.133+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:26.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:26.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:26.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:26.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:26.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:26.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:26.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:26.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:27.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:27.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:27.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:27.110+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:27.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:27.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:27.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:27.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:27.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:27.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:27.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:27.788+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:28.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:28.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:28.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:28.121+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:28.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:28.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:28.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:28.684+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:29.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:28.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:28.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:28.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:28.797+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:29.401 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:29.276+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36743 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:29.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:29.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:29.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:29.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:29.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:29.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:29.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:29.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:29.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:29.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:29.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:29.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:29.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:29.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:29.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:29.782+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:30.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:30.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:30.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:30.133+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:31.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:30.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:31.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:30.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:31.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:30.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:31.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:30.670+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:31.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:30.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:31.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:30.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:31.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:30.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:30.815+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:31.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:31.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:31.127+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:31.128+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:31.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:31.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:31.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:31.667+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:31.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:31.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:31.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:31.810+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:32.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:32.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:32.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:32.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:32.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:32.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:32.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:32.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:32.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:32.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:32.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:32.702+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:32.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:32.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:32.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:32.791+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:33.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:33.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:33.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:33.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:33.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:33.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:33.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:33.736+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:34.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:33.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:33.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:33.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:33.764+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:34.410 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:34.282+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36833 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:34.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:34.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:34.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:34.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:34.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:34.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:34.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:34.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:35.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:34.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:35.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:34.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:35.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:34.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:35.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:34.689+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:34.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:34.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:34.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:34.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:35.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:35.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:35.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:35.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:35.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:35.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:35.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:35.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:35.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:35.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:35.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:35.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:35.834+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:36.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:36.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:36.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:36.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:36.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:36.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:36.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:36.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:37.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:36.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:37.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:36.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:36.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:36.842+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:37.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:37.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:37.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:37.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:37.150+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:37.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:37.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:37.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:37.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:37.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:37.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:37.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:37.795+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:38.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:38.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:38.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:38.152+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:38.153+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:39.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:38.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:38.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:38.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:38.699+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:38.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:38.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:38.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:38.832+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:39.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:39.289+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 36923 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:39.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:39.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:39.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:39.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:39.183+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:39.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:39.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:39.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:39.693+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:40.200 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:39.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:40.200 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:39.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:40.200 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:39.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:40.200 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:39.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:40.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:40.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:40.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:40.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:40.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:40.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:40.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:40.648+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:40.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:40.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:40.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:40.807+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:41.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:41.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:41.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:41.168+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:42.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:41.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:41.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:41.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:41.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:41.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:41.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:41.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:41.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:42.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:42.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:42.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:42.205+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:42.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:42.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:42.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:42.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:42.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:42.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:42.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:42.800+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:43.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:43.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:43.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:43.213+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:43.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:43.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:43.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:43.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:43.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:43.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:43.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:43.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:44.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:44.295+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37014 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:44.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:44.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:44.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:44.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:44.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:44.220+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:44.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:44.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:44.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:44.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:44.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:44.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:44.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:44.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:45.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:45.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:45.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:45.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:45.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:45.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:45.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:46.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:45.566+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:46.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:45.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:46.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:45.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:46.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:45.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:46.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:45.755+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:46.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:46.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:46.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:46.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:46.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:46.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:46.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:46.297+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:46.788 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:46.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:46.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:46.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:46.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:46.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:46.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:46.518+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:46.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:46.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:46.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:46.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:47.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:47.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:47.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:47.284+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:47.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:47.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:47.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:47.507+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:48.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:47.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:48.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:47.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:48.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:47.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:48.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:47.799+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:48.236+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:48.236+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:48.236+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:48.237+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:48.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:48.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:48.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:48.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:48.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:48.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:48.486+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:49.207 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:48.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:49.207 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:48.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:49.207 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:48.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:49.207 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:48.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:49.498 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:49.302+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37101 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:49.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:49.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:49.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:49.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:49.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:49.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:49.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:49.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:49.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:49.513+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:49.513+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:49.513+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:49.514+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:50.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:49.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:50.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:49.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:50.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:49.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:50.189 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:49.858+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:50.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:50.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:50.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:50.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:50.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:50.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:50.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:50.188+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:50.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:50.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:50.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:50.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:50.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:50.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:50.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:50.486+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:51.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:50.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:51.183 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:50.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:51.183 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:50.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:51.183 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:50.863+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:51.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:51.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:51.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:51.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:51.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:51.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:51.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:51.182+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:51.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:51.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:51.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:51.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:51.519+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:52.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:51.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:52.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:51.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:52.233 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:51.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:52.233 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:51.876+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:52.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:52.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:52.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:52.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:52.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:52.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:52.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:52.232+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:52.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:52.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:52.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:52.522+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:53.211 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:52.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:53.211 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:52.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:53.211 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:52.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:53.211 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:52.913+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:53.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:53.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:53.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:53.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:53.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:53.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:53.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:53.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:53.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:53.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:53.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:53.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:53.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:53.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:53.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:53.560+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:54.239 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:53.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:54.239 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:53.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:54.239 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:53.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:54.240 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:53.931+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:54.513 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:54.308+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37191 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:41:54.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:54.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:54.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:54.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:54.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:54.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:54.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:54.239+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:54.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:54.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:54.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:54.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:54.513+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:55.231 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:54.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:55.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:54.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:55.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:54.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:55.232 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:54.958+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:55.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:55.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:55.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:55.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:55.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:55.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:55.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:55.231+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:55.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:55.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:55.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:55.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:56.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:56.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:56.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:56.224+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:55.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:55.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:55.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:55.976+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:57.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:56.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:57.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:56.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:57.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:56.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:57.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:56.542+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:57.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:57.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:57.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:57.265+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:57.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:57.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:57.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:57.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:57.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:57.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:57.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:57.022+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:58.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:57.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:58.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:57.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:58.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:57.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:58.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:57.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:58.007+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:58.007+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:58.007+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:58.007+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:58.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:58.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:58.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:58.230+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:58.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:58.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:58.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:41:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:58.630+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:59.316 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:59.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:59.317 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:59.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:59.317 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:59.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:41:59.317 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:41:59.262+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:41:59.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:59.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:59.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:59.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:59.317 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:59.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:41:59.318 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:41:59.049+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:41:59.587 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:41:59.315+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37280 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:00.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:59.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:00.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:59.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:00.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:59.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:00.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:41:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:41:59.587+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:00.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:00.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:00.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:00.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:00.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:00.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:00.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:00.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:00.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:00.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:00.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:00.097+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:01.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:00.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:01.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:00.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:01.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:00.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:01.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:00.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:01.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:01.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:01.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:01.070+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:01.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:01.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:01.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:01.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:01.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:01.214+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:02.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:01.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:02.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:01.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:02.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:01.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:02.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:01.607+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:02.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:02.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:02.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:02.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:02.202+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:02.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:02.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:02.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:02.035+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:03.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:02.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:03.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:02.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:03.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:02.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:03.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:02.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:03.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:03.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:03.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:03.187+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:03.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:03.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:03.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:03.035+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:03.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:03.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:03.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:03.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:04.335 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:04.321+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37370 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:04.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:04.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:04.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:04.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:04.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:04.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:04.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:04.171+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:04.336 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:04.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:04.336 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:04.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:04.336 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:04.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:04.337 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:04.078+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:05.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:04.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:04.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:04.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:04.663+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:05.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:05.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:05.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:05.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:05.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:05.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:05.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:05.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:05.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:05.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:05.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:05.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:05.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:05.071+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:06.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:05.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:06.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:05.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:06.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:05.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:06.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:05.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:06.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:06.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:06.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:06.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:06.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:06.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:06.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:06.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:06.041+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:06.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:06.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:06.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:06.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:07.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:07.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:07.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:07.078+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:07.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:07.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:07.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:07.062+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:07.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:07.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:07.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:07.724+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:08.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:08.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:08.086+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:08.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:08.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:08.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:08.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:08.099+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:08.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:08.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:08.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:08.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:09.455 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:09.328+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37461 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:09.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:09.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:09.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:09.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:09.456 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:09.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:09.456 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:09.114+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:09.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:09.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:09.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:09.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:09.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:09.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:09.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:09.104+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:09.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:09.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:09.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:09.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:10.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:10.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:10.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:10.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:10.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:10.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:10.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:10.134+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:10.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:10.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:10.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:10.154+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:11.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:10.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:11.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:10.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:11.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:10.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:11.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:10.641+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:11.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:11.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:11.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:11.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:11.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:11.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:11.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:11.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:11.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:11.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:11.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:11.093+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:11.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:11.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:11.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:11.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:12.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:12.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:12.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:12.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:12.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:12.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:12.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:12.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:12.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:12.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:12.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:12.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:12.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:12.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:12.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:12.069+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:12.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:12.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:12.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:12.644+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:13.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:13.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:13.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:13.116+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:13.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:13.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:13.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:13.098+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:13.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:13.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:13.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:13.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:14.336 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:14.054+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:14.336 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:14.054+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:14.337 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:14.054+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:14.337 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:14.055+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:14.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:14.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:14.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:14.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:14.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:14.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:14.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:14.094+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:14.637 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:14.335+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37551 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:15.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:14.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:15.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:14.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:15.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:14.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:15.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:14.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:15.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:15.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:15.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:15.030+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:15.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:15.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:15.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:15.097+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:15.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:15.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:15.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:15.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:15.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:15.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:15.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:15.621+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:16.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:16.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:16.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:16.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:15.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:15.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:15.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:15.991+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:16.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:16.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:16.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:16.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:16.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:16.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:16.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:16.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:17.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:17.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:17.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:17.062+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:16.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:16.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:16.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:16.983+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:17.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:17.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:17.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:17.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:17.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:17.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:17.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:17.689+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:18.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:18.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:18.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:18.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:17.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:17.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:17.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:17.975+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:18.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:18.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:18.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:18.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:18.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:18.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:18.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:18.655+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:19.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:19.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:19.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:19.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:18.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:18.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:18.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:18.991+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:19.613 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:19.341+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37640 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:19.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:19.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:19.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:19.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:19.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:19.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:19.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:19.613+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:20.246 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:20.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:20.246 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:20.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:20.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:20.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:20.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:20.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:20.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:19.955+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:20.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:19.955+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:20.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:19.955+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:20.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:19.956+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:20.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:20.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:20.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:20.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:20.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:20.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:20.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:20.662+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:20.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:20.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:20.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:20.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:20.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:20.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:20.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:20.960+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:21.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:21.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:21.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:21.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:21.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:21.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:21.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:21.669+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:21.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:21.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:21.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:21.975+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:21.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:22.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:21.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:22.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:21.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:22.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:21.913+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:22.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:22.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:22.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:22.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:22.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:22.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:22.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:22.904+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:22.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:22.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:22.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:22.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:22.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:22.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:22.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:22.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:23.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:22.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:22.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:22.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:22.940+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:23.973 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:23.858+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:23.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:23.858+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:23.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:23.858+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:23.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:23.859+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:23.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:23.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:23.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:23.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:23.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:23.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:23.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:23.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:24.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:23.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:23.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:23.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:23.973+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:24.655 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:24.348+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37721 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:24.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:24.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:24.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:24.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:24.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:24.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:24.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:24.885+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:24.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:24.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:24.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:24.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:24.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:24.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:24.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:24.655+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:25.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:24.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:24.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:24.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:24.952+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:25.912 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:25.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:25.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:25.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:25.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:25.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:25.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:25.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:25.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:25.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:25.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:25.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:25.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:25.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:25.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:25.904+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:25.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:25.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:25.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:25.912+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:27.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:26.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:26.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:26.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:26.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:27.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:26.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:27.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:26.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:27.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:26.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:27.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:26.927+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:26.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:26.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:26.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:26.864+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:28.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:27.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:27.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:27.734+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:27.735+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:28.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:27.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:27.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:27.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:27.923+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:27.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:27.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:27.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:27.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:29.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:28.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:28.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:28.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:28.694+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:29.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:28.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:29.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:28.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:28.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:28.865+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:28.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:28.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:28.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:28.920+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:29.682 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:29.355+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37811 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:29.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:29.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:29.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:29.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:29.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:29.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:29.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:29.904+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:29.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:29.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:29.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:29.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:29.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:29.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:29.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:29.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:30.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:29.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:30.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:29.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:30.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:29.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:30.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:29.943+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:30.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:30.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:30.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:30.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:30.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:30.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:30.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:30.646+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:30.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:30.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:30.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:30.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:30.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:30.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:30.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:30.893+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:31.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:30.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:30.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:30.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:30.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:32.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:31.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:31.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:31.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:31.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:32.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:31.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:32.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:31.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:32.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:31.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:32.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:31.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:31.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:31.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:31.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:31.896+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:32.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:32.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:32.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:32.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:32.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:32.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:32.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:32.837+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:32.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:32.673+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:32.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:32.673+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:32.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:32.673+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:32.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:32.674+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:33.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:32.926+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:32.926+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:32.926+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:32.926+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:33.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:33.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:33.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:33.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:33.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:33.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:33.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:33.672+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:33.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:33.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:33.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:33.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:33.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:33.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:33.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:33.791+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:34.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:33.924+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:33.924+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:33.924+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:33.925+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:34.659 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:34.361+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37897 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:34.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:34.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:34.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:34.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:34.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:34.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:34.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:34.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:34.916 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:34.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:34.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:34.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:34.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:34.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:34.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:34.781+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:35.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:34.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:34.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:34.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:34.915+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:35.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:35.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:35.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:35.695+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:35.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:36.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:35.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:36.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:35.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:36.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:35.820+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:35.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:35.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:35.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:36.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:35.890+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:36.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:36.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:36.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:36.722+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:36.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:36.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:36.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:36.823+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:36.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:36.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:36.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:36.929+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:38.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:37.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:37.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:37.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:37.750+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:37.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:37.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:37.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:37.911+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:37.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:37.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:37.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:37.790+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:38.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:38.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:38.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:38.719+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:38.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:38.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:38.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:38.912+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:38.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:38.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:38.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:38.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:39.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:39.367+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 37987 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:39.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:39.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:39.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:39.958+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:40.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:39.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:39.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:39.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:39.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:40.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:39.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:40.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:39.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:40.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:39.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:40.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:39.827+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:40.971+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:40.971+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:40.971+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:40.971+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:40.735+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:40.735+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:40.735+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:40.736+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:40.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:40.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:40.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:41.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:40.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:41.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:41.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:41.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:41.959+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:41.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:41.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:41.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:41.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:42.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:41.830+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:42.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:41.830+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:42.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:41.830+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:41.830+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:42.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:42.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:42.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:42.972+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:42.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:42.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:42.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:42.743+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:43.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:42.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:43.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:42.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:43.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:42.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:43.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:42.865+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:44.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:43.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:44.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:43.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:44.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:43.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:44.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:43.936+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:43.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:43.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:43.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:43.780+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:43.837+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:43.837+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:43.837+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:43.838+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:44.772 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:44.374+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38077 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:44.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:44.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:44.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:44.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:44.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:44.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:44.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:44.773+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:44.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:44.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:44.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:44.816+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:44.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:44.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:44.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:44.900+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:46.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:45.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:45.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:45.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:45.815+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:45.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:45.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:45.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:45.832+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:45.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:45.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:45.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:45.939+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:46.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:46.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:46.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:46.810+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:46.985+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:46.985+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:46.985+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:46.985+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:46.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:46.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:46.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:46.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:48.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:47.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:48.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:47.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:48.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:47.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:48.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:47.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:48.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:47.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:48.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:47.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:48.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:47.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:48.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:47.816+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:48.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:48.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:48.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:48.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:48.029+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:49.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:48.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:49.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:48.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:49.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:48.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:49.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:48.803+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:49.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:48.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:49.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:48.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:49.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:48.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:49.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:48.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:49.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:49.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:49.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:49.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:49.022+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:49.747 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:49.381+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38167 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:50.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:49.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:50.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:49.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:50.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:49.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:50.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:49.819+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:50.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:49.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:50.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:49.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:50.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:49.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:50.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:49.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:50.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:50.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:50.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:50.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:50.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:50.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:50.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:50.023+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:50.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:50.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:50.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:50.732+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:50.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:50.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:50.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:50.846+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:50.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:50.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:50.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:50.976+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:51.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:51.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:51.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:51.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:51.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:51.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:51.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:51.841+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:51.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:51.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:51.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:51.969+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:52.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:52.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:52.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:52.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:52.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:52.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:53.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:52.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:53.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:52.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:52.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:52.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:52.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:52.972+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:54.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:53.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:54.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:53.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:54.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:53.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:54.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:53.791+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:53.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:53.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:53.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:54.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:53.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:54.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:54.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:54.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:54.005+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:54.776 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:54.387+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38250 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:42:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:54.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:54.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:54.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:54.873+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:54.994+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:54.994+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:54.994+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:55.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:54.995+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:54.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:54.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:54.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:54.776+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:55.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:55.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:55.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:55.869+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:55.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:55.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:55.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:55.961+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:55.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:55.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:55.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:56.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:55.806+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:57.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:56.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:57.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:56.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:57.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:56.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:57.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:56.961+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:56.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:56.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:56.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:57.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:56.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:56.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:56.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:56.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:56.780+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:58.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:57.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:57.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:57.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:57.959+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:57.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:57.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:57.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:57.795+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:58.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:57.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:58.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:57.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:58.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:57.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:58.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:57.813+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:59.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:58.935+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:58.935+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:58.935+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:42:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:58.936+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:58.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:58.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:59.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:58.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:42:59.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:58.837+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:42:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:58.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:58.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:58.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:42:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:58.781+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:42:59.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:42:59.394+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38339 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:59.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:59.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:59.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:42:59.964+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:59.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:59.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:59.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:42:59.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:00.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:59.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:00.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:59.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:00.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:59.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:00.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:42:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:42:59.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:01.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:00.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:00.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:00.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:00.939+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:01.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:00.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:01.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:00.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:00.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:00.722+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:01.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:00.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:01.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:00.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:01.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:00.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:01.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:00.934+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:01.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:01.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:01.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:01.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:01.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:01.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:01.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:01.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:02.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:01.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:01.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:01.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:01.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:01.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:01.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:01.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:01.952+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:02.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:02.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:02.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:02.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:02.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:02.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:02.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:02.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:02.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:02.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:02.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:02.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:02.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:02.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:02.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:02.959+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:03.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:03.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:03.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:03.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:03.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:03.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:03.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:03.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:03.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:03.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:03.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:03.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:03.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:03.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:03.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:03.920+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:03.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:03.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:03.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:03.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:04.666 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:04.401+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38429 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:04.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:04.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:04.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:04.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:04.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:04.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:04.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:04.666+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:04.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:04.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:04.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:04.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:04.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:04.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:04.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:04.875+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:04.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:04.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:04.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:04.936+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:05.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:05.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:05.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:05.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:05.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:05.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:05.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:05.904+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:05.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:05.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:05.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:05.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:05.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:05.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:05.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:05.666+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:05.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:05.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:05.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:05.932+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:06.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:06.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:06.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:06.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:06.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:06.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:06.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:06.676+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:06.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:06.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:06.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:06.952+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:06.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:06.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:06.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:07.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:06.954+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:07.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:07.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:07.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:07.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:08.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:07.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:07.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:07.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:07.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:07.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:07.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:07.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:08.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:07.954+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:08.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:08.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:08.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:08.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:08.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:08.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:08.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:08.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:08.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:08.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:08.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:08.949+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:08.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:08.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:08.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:08.997+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:09.705 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:09.407+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38520 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:09.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:09.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:09.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:09.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:09.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:09.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:09.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:09.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:09.984 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:09.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:09.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:09.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:09.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:09.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:09.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:09.926+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:10.248 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:09.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:10.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:09.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:10.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:09.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:10.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:09.983+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:10.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:10.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:10.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:10.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:10.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:10.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:10.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:10.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:10.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:10.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:10.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:10.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:10.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:10.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:10.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:10.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:11.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:10.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:10.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:10.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:10.977+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:12.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:11.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:12.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:11.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:11.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:11.695+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:11.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:11.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:11.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:11.934+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:11.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:11.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:11.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:11.861+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:13.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:12.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:12.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:12.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:12.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:12.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:12.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:12.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:12.847+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:12.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:12.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:12.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:12.898+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:13.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:13.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:13.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:13.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:13.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:13.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:13.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:13.823+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:13.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:13.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:13.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:13.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:13.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:13.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:13.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:13.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:14.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:13.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:13.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:13.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:13.926+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:14.703 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:14.414+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38610 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:15.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:14.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:14.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:14.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:14.929+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:14.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:14.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:14.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:14.703+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:14.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:14.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:14.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:14.829+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:15.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:15.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:15.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:15.916+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:15.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:15.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:16.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:15.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:16.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:15.741+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:16.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:15.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:16.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:15.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:16.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:15.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:16.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:15.807+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:17.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:16.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:16.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:16.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:16.958+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:16.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:16.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:16.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:16.787+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:16.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:16.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:16.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:16.809+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:18.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:17.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:18.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:17.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:18.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:17.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:18.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:17.928+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:17.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:17.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:17.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:17.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:17.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:17.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:17.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:17.773+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:19.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:18.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:18.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:18.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:18.929+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:18.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:18.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:18.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:18.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:19.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:18.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:19.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:18.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:19.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:18.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:19.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:18.754+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:19.775 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:19.420+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38700 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:20.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:19.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:20.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:19.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:20.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:19.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:20.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:19.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:20.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:19.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:20.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:19.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:20.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:19.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:20.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:19.792+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:20.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:19.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:20.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:19.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:20.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:19.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:20.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:19.896+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:21.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:20.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:20.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:20.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:20.783+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:21.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:20.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:21.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:20.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:21.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:20.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:21.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:20.815+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:21.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:20.923+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:21.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:20.923+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:21.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:20.923+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:21.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:20.923+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:22.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:21.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:21.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:21.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:21.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:22.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:21.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:22.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:21.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:22.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:21.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:21.770+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:21.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:21.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:21.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:21.882+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:23.089 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:22.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:23.089 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:22.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:23.089 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:22.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:23.089 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:22.832+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:23.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:22.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:23.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:22.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:23.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:22.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:23.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:22.904+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:23.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:22.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:23.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:22.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:23.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:22.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:23.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:22.802+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:23.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:23.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:23.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:23.803+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:23.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:23.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:23.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:23.832+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:23.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:23.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:23.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:23.949+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:24.771 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:24.427+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38790 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:24.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:25.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:24.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:25.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:24.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:25.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:24.804+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:24.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:24.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:24.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:24.900+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:24.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:24.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:24.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:24.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:26.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:25.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:25.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:25.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:25.727+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:25.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:26.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:25.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:26.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:25.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:26.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:25.768+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:25.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:25.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:25.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:26.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:25.852+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:27.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:26.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:26.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:26.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:26.690+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:27.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:26.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:27.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:26.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:27.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:26.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:27.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:26.778+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:26.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:26.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:26.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:26.850+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:28.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:27.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:27.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:27.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:27.693+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:28.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:27.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:28.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:27.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:27.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:27.745+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:27.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:27.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:27.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:27.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:29.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:28.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:28.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:28.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:28.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:29.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:28.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:29.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:28.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:29.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:28.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:28.725+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:28.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:28.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:28.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:28.847+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:29.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:29.434+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38880 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:30.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:29.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:30.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:29.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:30.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:29.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:30.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:29.839+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:29.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:30.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:29.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:30.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:29.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:30.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:29.740+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:30.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:29.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:30.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:29.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:30.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:29.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:30.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:29.771+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:31.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:30.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:31.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:30.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:31.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:30.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:31.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:30.821+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:31.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:30.846+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:30.846+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:30.846+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:30.846+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:31.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:30.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:31.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:30.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:31.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:30.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:31.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:30.691+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:32.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:31.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:31.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:31.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:31.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:32.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:31.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:32.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:31.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:32.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:31.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:32.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:31.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:31.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:31.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:31.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:31.828+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:32.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:33.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:32.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:33.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:32.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:33.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:32.847+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:32.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:32.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:32.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:33.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:32.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:32.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:32.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:32.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:32.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:33.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:33.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:33.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:33.834+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:33.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:33.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:33.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:33.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:33.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:33.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:33.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:34.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:33.873+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:34.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:34.442+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 38969 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:34.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:34.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:34.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:34.645+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:35.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:34.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:34.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:34.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:34.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:34.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:34.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:34.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:34.830+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:35.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:35.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:35.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:35.872+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:35.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:35.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:35.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:35.662+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:35.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:35.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:35.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:35.813+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:36.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:36.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:36.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:36.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:36.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:36.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:36.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:36.629+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:36.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:36.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:36.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:36.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:37.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:37.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:37.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:37.824+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:37.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:37.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:37.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:37.620+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:37.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:37.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:37.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:37.791+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:38.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:38.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:38.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:38.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:38.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:38.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:38.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:38.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:38.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:38.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:38.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:38.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:38.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:38.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:38.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:38.634+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:38.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:38.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:38.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:38.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:38.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:38.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:38.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:38.795+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:39.755 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:39.448+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39060 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:39.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:39.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:39.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:39.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:39.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:39.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:39.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:39.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:40.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:39.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:39.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:39.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:39.755+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:39.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:39.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:39.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:39.768+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:41.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:40.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:40.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:40.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:40.775+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:40.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:40.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:40.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:40.753+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:40.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:40.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:40.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:40.631+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:41.771 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:41.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:41.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:41.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:41.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:41.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:41.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:41.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:41.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:41.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:41.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:41.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:41.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:41.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:41.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:41.711+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:42.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:41.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:41.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:41.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:41.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:42.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:42.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:42.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:42.664+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:42.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:42.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:42.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:42.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:42.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:42.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:42.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:42.603+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:44.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:43.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:44.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:43.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:43.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:43.573+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:43.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:43.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:43.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:43.707+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:43.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:43.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:43.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:43.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:44.711 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:44.455+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39148 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:44.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:44.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:44.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:44.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:44.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:44.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:44.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:44.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:45.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:44.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:44.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:44.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:44.743+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:44.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:44.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:44.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:44.710+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:45.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:45.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:45.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:45.712+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:46.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:45.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:46.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:45.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:45.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:45.587+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:45.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:45.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:45.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:45.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:47.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:46.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:46.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:46.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:46.575+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:46.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:46.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:46.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:47.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:46.725+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:46.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:46.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:46.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:46.721+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:47.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:47.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:47.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:47.748+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:47.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:47.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:47.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:47.751+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:47.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:47.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:47.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:47.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:49.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:48.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:48.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:48.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:48.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:48.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:48.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:48.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:48.755+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:48.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:48.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:48.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:48.751+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:49.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:49.462+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39239 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:49.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:49.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:49.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:49.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:49.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:49.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:49.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:49.559+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:50.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:49.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:49.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:49.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:49.728+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:49.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:49.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:49.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:50.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:49.721+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:51.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:50.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:50.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:50.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:50.547+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:50.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:50.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:50.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:50.699+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:50.737+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:51.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:51.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:51.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:51.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:51.532+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:51.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:51.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:51.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:51.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:51.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:51.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:51.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:51.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:52.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:52.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:52.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:52.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:52.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:52.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:52.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:52.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:52.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:52.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:52.776+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:52.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:52.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:52.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:52.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:53.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:53.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:53.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:53.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:53.780+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:53.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:53.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:53.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:53.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:53.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:53.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:53.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:53.534+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:54.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:54.468+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39327 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:54.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:54.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:54.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:54.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:54.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:54.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:54.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:54.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:54.498+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:55.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:54.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:54.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:54.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:54.826+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:55.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:55.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:55.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:55.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:55.695+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:55.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:55.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:55.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:55.450+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:55.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:55.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:55.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:55.846+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:56.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:56.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:56.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:56.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:56.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:56.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:56.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:56.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:56.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:56.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:56.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:56.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:56.862+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:57.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:57.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:57.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:57.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:57.496+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:57.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:57.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:57.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:57.663+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:57.850+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:57.850+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:57.850+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:57.851+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:58.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:58.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:58.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:58.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:58.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:58.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:58.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:58.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:58.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:43:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:58.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:43:59.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:58.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:58.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:58.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:43:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:58.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:43:59.715 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:43:59.474+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39417 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:43:59.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:59.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:59.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:59.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:59.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:59.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:43:59.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:43:59.438+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:00.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:00.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:43:59.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:59.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:59.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:59.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:43:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:43:59.848+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:00.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:00.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:00.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:00.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:00.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:00.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:00.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:00.426+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:00.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:00.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:00.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:00.876+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:00.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:00.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:00.735+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:00.736+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:01.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:01.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:01.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:01.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:01.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:01.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:01.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:01.395+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:02.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:01.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:01.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:01.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:01.908+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:01.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:01.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:01.761+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:01.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:02.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:02.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:02.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:02.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:02.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:02.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:02.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:02.409+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:02.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:02.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:02.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:02.946+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:02.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:02.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:02.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:02.782+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:03.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:03.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:03.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:03.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:03.447+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:03.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:03.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:03.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:03.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:03.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:03.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:03.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:03.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:04.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:04.481+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39506 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:04.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:04.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:04.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:04.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:05.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:05.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:05.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:05.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:04.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:04.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:04.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:04.821+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:05.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:05.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:05.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:05.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:05.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:06.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:06.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:06.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:06.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:05.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:05.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:05.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:05.827+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:06.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:06.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:06.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:06.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:06.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:06.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:06.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:06.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:06.957+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:06.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:06.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:06.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:07.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:06.828+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:07.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:07.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:07.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:07.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:07.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:07.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:08.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:07.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:07.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:07.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:07.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:07.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:07.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:07.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:07.813+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:08.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:08.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:08.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:08.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:08.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:08.397+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:08.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:08.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:08.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:08.966+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:08.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:08.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:08.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:08.856+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:09.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:09.488+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39598 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:09.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:09.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:09.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:09.400+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:10.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:10.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:10.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:10.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:10.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:10.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:09.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:10.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:09.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:10.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:09.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:10.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:09.882+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:10.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:10.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:10.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:10.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:10.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:10.395+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:10.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:10.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:10.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:10.990+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:10.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:10.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:10.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:11.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:10.869+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:11.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:11.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:11.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:11.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:11.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:12.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:12.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:12.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:12.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:11.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:11.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:11.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:11.905+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:12.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:12.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:12.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:12.387+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:12.388+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:13.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:13.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:13.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:13.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:12.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:12.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:12.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:12.874+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:13.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:13.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:13.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:13.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:13.419+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:14.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:14.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:14.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:14.091+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:13.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:13.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:14.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:13.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:14.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:13.850+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:14.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:14.495+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39687 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:14.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:14.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:14.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:14.412+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:15.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:14.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:15.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:14.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:15.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:14.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:15.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:14.813+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:15.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:15.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:15.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:15.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:15.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:15.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:15.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:15.112+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:15.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:15.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:15.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:15.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:15.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:15.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:15.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:15.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:16.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:15.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:16.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:15.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:15.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:15.778+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:16.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:16.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:16.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:16.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:16.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:16.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:16.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:16.109+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:16.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:16.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:16.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:16.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:16.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:16.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:16.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:16.410+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:17.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:16.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:17.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:16.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:16.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:16.754+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:17.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:17.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:17.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:17.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:17.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:17.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:17.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:17.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:17.153+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:18.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:17.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:18.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:17.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:18.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:17.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:18.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:17.742+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:18.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:18.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:18.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:18.391+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:18.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:18.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:18.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:18.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:18.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:18.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:18.185+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:19.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:18.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:19.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:18.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:18.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:18.729+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:19.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:19.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:19.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:19.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:19.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:19.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:19.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:19.440+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:19.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:19.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:19.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:19.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:19.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:19.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:19.502+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39778 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:19.704+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:19.704+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:19.704+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:19.704+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:20.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:20.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:20.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:20.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:20.448+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:20.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:20.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:20.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:20.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:21.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:20.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:20.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:20.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:20.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:21.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:21.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:21.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:21.463+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:21.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:21.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:21.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:21.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:22.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:21.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:22.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:21.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:21.712+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:21.713+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:22.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:22.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:22.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:22.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:22.456+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:22.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:22.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:22.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:22.232+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:23.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:22.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:22.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:22.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:22.699+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:23.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:23.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:23.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:23.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:23.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:23.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:23.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:23.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:23.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:23.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:23.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:23.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:23.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:23.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:23.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:23.663+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:23.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:23.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:23.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:23.476+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:24.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:24.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:24.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:24.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:24.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:24.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:24.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:24.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:24.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:24.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:24.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:24.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:24.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:24.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:24.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:24.235+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:24.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:24.509+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39867 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:24.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:24.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:24.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:24.640+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:25.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:25.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:25.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:25.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:25.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:25.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:25.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:25.513+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:25.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:25.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:25.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:25.284+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:26.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:25.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:26.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:25.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:26.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:25.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:26.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:25.611+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:26.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:26.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:26.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:26.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:26.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:26.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:26.558 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:26.294+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:27.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:26.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:27.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:26.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:27.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:26.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:26.558+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:27.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:26.594+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:27.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:26.594+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:27.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:26.594+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:27.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:26.594+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:27.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:27.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:27.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:27.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:27.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:27.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:27.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:27.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:27.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:27.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:27.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:27.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:27.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:27.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:27.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:27.531+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:27.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:27.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:27.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:27.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:27.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:27.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:27.596+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:28.623 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:28.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:28.623 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:28.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:28.623 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:28.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:28.623 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:28.515+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:28.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:28.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:28.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:28.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:28.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:28.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:28.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:28.293+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:29.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:28.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:29.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:28.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:29.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:28.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:29.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:28.622+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:29.591 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:29.516+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 39959 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:29.592 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:29.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:29.592 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:29.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:29.592 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:29.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:29.592 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:29.294+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:29.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:29.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:29.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:29.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:29.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:29.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:29.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:29.494+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:30.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:29.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:30.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:29.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:30.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:29.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:30.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:29.591+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:30.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:30.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:30.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:30.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:30.457+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:30.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:30.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:30.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:30.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:30.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:30.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:30.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:30.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:30.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:30.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:30.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:30.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:31.591 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:31.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:31.591 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:31.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:31.591 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:31.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:31.591 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:31.299+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:31.591 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:31.434+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:31.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:31.434+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:31.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:31.434+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:31.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:31.435+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:32.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:31.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:32.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:31.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:31.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:31.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:32.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:32.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:32.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:32.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:32.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:32.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:32.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:32.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:32.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:32.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:32.289+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:33.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:32.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:33.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:32.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:32.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:32.625+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:33.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:33.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:33.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:33.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:33.408+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:33.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:33.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:33.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:33.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:33.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:33.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:33.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:33.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:33.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:33.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:33.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:33.602+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:34.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:34.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:34.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:34.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:34.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:34.362+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:34.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:34.362+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:34.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:34.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:34.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:34.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:34.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:34.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:34.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:34.239+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:34.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:34.522+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40048 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:34.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:34.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:34.619+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:34.619+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:35.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:35.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:35.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:35.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:35.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:35.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:35.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:35.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:36.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:35.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:36.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:35.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:35.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:35.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:36.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:36.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:36.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:36.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:36.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:36.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:36.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:36.327+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:36.710 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:36.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:36.710 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:36.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:36.710 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:36.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:36.710 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:36.303+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:37.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:36.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:37.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:36.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:37.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:36.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:37.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:36.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:37.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:37.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:37.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:37.286+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:37.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:37.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:37.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:37.288+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:38.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:37.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:37.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:37.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:37.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:38.320 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:38.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:38.320 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:38.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:38.320 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:38.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:38.320 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:38.275+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:38.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:38.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:38.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:38.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:38.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:38.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:38.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:38.320+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:39.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:39.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:38.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:39.654 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:39.529+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40139 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:39.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:39.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:39.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:39.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:39.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:39.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:39.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:39.312+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:39.655 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:39.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:39.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:39.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:39.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:39.336+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:39.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:39.336+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:39.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:39.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:39.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:39.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:40.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:40.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:40.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:40.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:40.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:40.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:40.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:40.340+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:40.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:40.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:40.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:40.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:40.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:40.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:40.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:40.303+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:41.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:40.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:41.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:40.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:41.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:40.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:40.691+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:41.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:41.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:41.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:41.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:41.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:41.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:41.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:41.300+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:41.716 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:41.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:41.716 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:41.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:41.716 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:41.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:41.716 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:41.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:42.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:41.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:41.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:41.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:41.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:42.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:42.327+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:42.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:42.327+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:42.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:42.327+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:42.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:42.328+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:42.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:42.344+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:42.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:42.344+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:42.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:42.344+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:42.345+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:43.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:42.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:43.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:42.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:42.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:42.748+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:43.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:43.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:43.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:43.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:43.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:43.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:43.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:43.366+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:43.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:43.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:43.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:43.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:43.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:43.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:43.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:43.299+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:44.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:43.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:44.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:43.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:44.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:43.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:44.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:43.701+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:44.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:44.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:44.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:44.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:44.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:44.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:44.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:44.371+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:44.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:44.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:44.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:44.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:44.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:44.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:44.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:44.270+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:44.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:44.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:44.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:44.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:44.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:44.681+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:44.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:44.534+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40228 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:45.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:45.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:45.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:45.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:45.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:45.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:45.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:45.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:45.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:45.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:45.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:45.222+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:46.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:45.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:46.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:45.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:45.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:45.717+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:46.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:46.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:46.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:46.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:46.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:46.459+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:46.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:46.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:46.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:46.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:46.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:46.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:46.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:46.222+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:47.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:46.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:46.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:46.755+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:46.755+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:47.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:47.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:47.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:47.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:47.234+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:47.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:47.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:47.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:47.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:47.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:47.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:48.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:47.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:47.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:47.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:47.784+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:48.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:48.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:48.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:48.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:48.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:48.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:48.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:48.276+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:49.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:48.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:49.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:48.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:48.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:48.812+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:49.562 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:49.541+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40320 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:49.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:49.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:49.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:49.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:49.443+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:49.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:49.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:49.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:49.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:49.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:49.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:49.564 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:49.305+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:50.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:49.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:50.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:49.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:49.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:49.797+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:50.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:50.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:50.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:50.397+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:50.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:50.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:50.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:50.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:51.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:50.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:51.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:50.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:51.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:50.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:51.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:50.811+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:51.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:51.236+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:51.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:51.236+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:51.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:51.236+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:51.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:51.237+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:51.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:51.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:51.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:51.398+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:51.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:52.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:51.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:52.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:51.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:52.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:51.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:52.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:52.366+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:52.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:52.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:52.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:52.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:52.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:52.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:52.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:52.284+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:52.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:52.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:53.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:52.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:53.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:52.786+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:53.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:53.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:53.747 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:53.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:53.747 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:53.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:53.747 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:53.331+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:53.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:53.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:53.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:53.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:53.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:53.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:53.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:53.325+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:53.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:53.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:53.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:53.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:54.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:54.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:54.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:54.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:54.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:54.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:54.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:54.333+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:54.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:54.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:54.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:54.288+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:55.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:54.548+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40409 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:55.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:54.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:54.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:54.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:54.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:55.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:55.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:55.723 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:55.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:55.723 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:55.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:55.723 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:55.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:55.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:55.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:55.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:55.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:55.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:55.301+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:55.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:55.301+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:56.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:55.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:56.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:55.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:56.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:55.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:56.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:55.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:56.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:56.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:56.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:56.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:56.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:56.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:56.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:56.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:56.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:56.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:56.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:56.288+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:57.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:56.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:57.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:56.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:57.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:56.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:57.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:56.705+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:57.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:57.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:57.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:57.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:57.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:57.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:57.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:57.339+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:57.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:57.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:57.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:57.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:57.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:57.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:57.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:57.338+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:58.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:57.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:58.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:57.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:58.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:57.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:58.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:57.705+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:58.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:58.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:58.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:58.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:58.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:58.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:58.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:58.295+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:58.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:58.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:58.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:58.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:58.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:58.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:58.665 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:58.319+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:44:59.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:58.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:59.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:58.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:59.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:58.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:44:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:58.663+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:59.628 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:44:59.555+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40500 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:44:59.629 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:59.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:59.629 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:59.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:59.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:59.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:44:59.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:44:59.340+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:44:59.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:59.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:59.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:59.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:59.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:59.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:44:59.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:44:59.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:00.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:59.628+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:00.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:59.628+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:59.628+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:44:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:44:59.628+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:00.623 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:00.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:00.623 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:00.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:00.623 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:00.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:00.623 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:00.358+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:00.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:00.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:00.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:00.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:00.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:00.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:00.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:00.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:01.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:00.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:01.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:00.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:01.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:00.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:01.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:00.624+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:01.642 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:01.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:01.643 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:01.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:01.643 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:01.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:01.643 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:01.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:01.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:01.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:01.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:01.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:01.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:01.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:01.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:01.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:02.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:01.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:02.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:01.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:02.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:01.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:01.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:02.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:02.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:02.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:02.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:02.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:02.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:02.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:02.362+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:02.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:02.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:02.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:02.278+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:03.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:02.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:02.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:02.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:02.664+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:03.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:03.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:03.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:03.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:03.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:03.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:03.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:03.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:03.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:03.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:03.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:03.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:03.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:03.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:03.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:03.381+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:04.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:03.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:04.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:03.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:03.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:03.668+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:04.675 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:04.562+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40589 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:04.675 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:04.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:04.675 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:04.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:04.675 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:04.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:04.676 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:04.341+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:04.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:04.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:04.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:04.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:04.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:04.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:04.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:04.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:05.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:04.674+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:05.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:04.674+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:05.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:04.674+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:05.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:04.675+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:05.677 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:05.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:05.677 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:05.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:05.677 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:05.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:05.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:05.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:05.678 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:05.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:05.678 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:05.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:05.678 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:05.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:05.678 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:05.361+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:06.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:05.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:06.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:05.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:05.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:05.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:06.644 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:06.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:06.645 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:06.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:06.645 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:06.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:06.645 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:06.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:06.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:06.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:06.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:06.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:06.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:06.321+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:06.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:06.321+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:07.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:06.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:07.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:06.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:07.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:06.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:06.644+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:07.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:07.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:07.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:07.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:07.289+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:07.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:07.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:07.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:07.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:08.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:07.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:08.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:07.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:08.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:07.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:07.602+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:08.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:08.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:08.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:08.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:08.280+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:08.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:08.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:08.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:08.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:08.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:08.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:08.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:08.260+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:09.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:08.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:09.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:08.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:09.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:08.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:09.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:08.581+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:09.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:09.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:09.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:09.245+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:09.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:09.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:09.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:09.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:09.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:09.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:09.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:09.276+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:09.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:09.569+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40681 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:09.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:09.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:09.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:09.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:09.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:09.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:10.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:10.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:10.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:10.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:10.248+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:10.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:10.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:10.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:10.202+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:11.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:10.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:11.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:10.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:11.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:10.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:11.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:10.567+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:11.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:11.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:11.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:11.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:11.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:11.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:11.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:11.233+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:11.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:11.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:11.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:11.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:12.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:11.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:12.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:11.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:12.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:11.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:12.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:11.592+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:12.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:12.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:12.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:12.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:12.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:12.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:12.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:12.151+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:13.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:12.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:13.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:12.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:13.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:12.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:13.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:12.558+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:13.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:13.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:13.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:13.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:13.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:13.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:13.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:13.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:13.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:13.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:13.155+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:14.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:13.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:14.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:13.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:14.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:13.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:14.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:13.547+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:14.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:14.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:14.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:14.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:14.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:14.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:14.268+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:14.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:14.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:14.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:14.127+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:15.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:14.576+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40770 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:15.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:14.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:15.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:14.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:14.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:14.548+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:15.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:15.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:15.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:15.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:15.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:15.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:15.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:15.132+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:15.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:15.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:15.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:15.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:15.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:15.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:15.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:15.306+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:15.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:15.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:15.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:15.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:15.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:15.521+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:16.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:16.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:16.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:16.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:16.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:16.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:16.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:16.110+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:16.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:16.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:16.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:16.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:16.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:16.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:16.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:16.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:16.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:16.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:16.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:16.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:16.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:16.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:16.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:16.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:17.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:17.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:17.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:17.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:17.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:17.144+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:17.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:17.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:17.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:17.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:18.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:17.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:18.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:17.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:18.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:17.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:18.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:17.575+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:18.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:18.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:18.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:18.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:18.187+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:18.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:18.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:18.199+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:18.200+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:19.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:18.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:19.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:18.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:19.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:18.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:18.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:19.154+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:19.154+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:19.154+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:19.155+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:19.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:19.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:19.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:19.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:19.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:19.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:19.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:19.193+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:19.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:19.583+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40861 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:19.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:19.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:19.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:19.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:19.622+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:20.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:20.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:20.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:20.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:20.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:20.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:20.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:20.235+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:20.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:20.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:20.108+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:20.109+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:21.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:20.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:21.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:20.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:21.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:20.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:20.616+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:21.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:21.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:21.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:21.198+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:21.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:21.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:21.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:21.111+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:22.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:21.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:22.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:21.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:21.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:21.602+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:22.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:22.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:22.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:22.152+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:22.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:22.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:22.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:22.062+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:23.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:22.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:23.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:22.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:23.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:22.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:23.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:22.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:23.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:23.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:23.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:23.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:23.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:23.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:23.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:23.014+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:23.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:23.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:23.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:23.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:24.011 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:23.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:24.011 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:23.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:24.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:23.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:24.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:23.613+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:24.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:24.078+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:24.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:24.078+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:24.078+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:24.078+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:24.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:24.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:24.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:24.011+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:24.970 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:24.590+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 40950 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:24.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:24.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:24.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:24.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:24.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:24.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:24.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:24.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:25.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:25.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:25.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:25.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:25.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:25.082+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:24.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:24.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:24.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:24.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:25.968 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:25.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:25.968 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:25.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:25.968 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:25.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:25.968 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:25.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:26.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:26.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:26.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:26.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:26.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:26.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:26.117+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:25.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:25.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:25.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:25.968+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:27.006 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:26.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:27.006 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:26.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:27.006 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:26.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:27.006 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:26.693+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:27.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:27.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:27.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:27.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:27.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:27.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:27.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:27.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:27.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:27.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:27.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:27.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:27.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:27.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:27.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:27.979 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:27.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:28.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:27.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:28.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:27.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:28.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:27.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:28.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:27.978+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:28.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:28.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:28.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:28.061+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:28.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:28.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:28.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:28.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:28.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:28.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:28.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:28.697+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:29.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:28.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:29.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:28.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:29.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:28.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:28.984+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:29.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:29.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:29.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:29.034+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:29.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:29.597+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41042 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:29.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:29.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:29.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:29.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:29.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:29.731+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:30.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:30.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:30.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:30.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:30.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:30.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:30.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:30.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:30.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:30.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:30.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:30.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:30.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:30.083+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:30.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:30.084+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:30.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:30.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:30.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:30.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:30.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:30.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:30.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:30.705+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:31.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:31.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:31.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:31.063+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:30.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:30.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:30.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:30.978+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:31.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:31.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:31.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:31.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:31.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:31.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:31.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:31.684+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:32.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:32.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:32.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:32.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:32.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:32.106+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:31.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:31.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:31.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:31.942+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:32.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:32.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:32.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:32.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:32.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:32.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:32.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:32.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:33.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:33.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:33.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:33.138+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:32.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:32.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:32.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:32.942+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:33.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:33.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:33.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:33.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:33.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:33.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:33.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:33.673+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:34.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:34.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:34.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:34.144+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:33.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:33.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:33.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:33.982+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:34.951 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:34.604+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41130 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:34.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:34.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:34.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:34.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:34.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:34.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:34.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:34.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:35.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:35.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:35.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:35.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:35.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:35.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:35.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:35.141+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:34.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:34.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:34.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:34.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:36.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:35.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:36.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:35.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:35.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:35.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:35.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:35.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:35.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:36.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:35.956+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:36.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:36.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:36.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:36.157+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:37.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:36.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:37.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:36.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:37.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:36.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:37.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:36.740+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:36.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:36.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:36.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:36.907+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:37.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:37.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:37.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:37.149+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:38.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:37.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:38.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:37.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:38.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:37.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:38.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:37.785+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:37.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:37.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:37.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:37.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:38.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:38.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:38.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:38.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:38.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:38.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:38.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:38.151+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:39.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:38.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:39.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:38.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:39.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:38.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:39.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:38.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:39.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:38.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:39.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:38.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:39.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:38.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:39.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:38.823+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:39.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:39.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:39.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:39.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:39.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:39.129+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:39.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:39.612+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41221 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:40.110 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:39.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:40.110 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:39.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:40.111 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:39.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:40.111 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:39.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:40.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:39.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:40.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:39.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:40.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:39.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:40.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:39.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:40.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:40.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:40.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:40.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:40.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:40.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:40.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:40.111+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:41.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:40.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:41.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:40.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:41.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:40.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:41.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:40.891+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:41.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:40.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:41.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:40.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:41.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:40.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:41.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:40.861+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:41.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:41.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:41.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:41.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:41.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:41.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:41.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:41.146+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:42.168 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:41.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:42.168 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:41.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:42.168 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:41.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:42.168 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:41.940+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:42.168 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:41.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:42.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:41.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:42.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:41.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:42.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:41.881+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:42.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:42.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:42.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:42.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:42.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:42.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:42.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:42.168+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:43.208 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:42.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:43.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:42.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:43.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:42.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:43.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:42.915+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:43.209 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:42.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:43.209 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:42.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:43.209 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:42.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:43.209 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:42.877+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:43.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:43.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:43.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:43.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:43.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:43.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:43.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:43.209+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:44.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:43.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:44.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:43.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:44.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:43.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:44.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:43.903+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:44.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:43.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:44.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:43.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:44.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:43.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:44.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:43.914+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:44.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:44.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:44.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:44.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:44.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:44.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:44.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:44.246+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:44.869 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:44.619+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41310 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:45.216 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:44.869+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:45.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:44.869+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:45.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:44.869+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:45.217 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:44.869+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:45.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:44.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:45.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:44.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:45.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:44.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:45.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:44.892+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:45.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:45.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:45.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:45.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:45.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:45.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:45.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:45.217+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:46.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:45.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:46.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:45.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:46.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:45.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:46.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:45.834+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:46.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:45.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:46.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:45.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:46.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:45.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:46.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:45.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:46.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:46.220+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:46.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:46.220+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:46.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:46.220+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:46.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:46.221+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:47.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:46.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:47.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:46.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:47.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:46.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:47.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:46.863+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:47.259 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:46.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:47.259 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:46.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:47.259 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:46.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:47.259 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:46.877+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:47.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:47.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:47.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:47.259+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:48.220 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:47.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:48.220 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:47.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:48.220 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:47.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:48.220 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:47.842+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:48.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:47.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:48.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:47.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:48.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:47.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:48.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:47.834+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:48.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:48.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:48.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:48.220+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:49.243 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:48.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:49.243 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:48.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:49.243 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:48.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:49.243 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:48.832+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:49.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:48.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:49.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:48.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:49.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:48.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:49.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:48.817+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:49.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:49.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:49.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:49.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:49.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:49.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:49.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:49.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:49.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:49.625+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41402 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:50.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:49.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:50.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:49.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:50.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:49.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:49.864+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:49.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:49.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:49.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:49.773+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:50.236+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:50.236+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:50.236+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:50.237+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:50.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:50.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:50.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:51.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:50.830+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:50.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:50.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:50.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:50.726+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:51.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:51.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:51.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:51.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:51.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:51.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:51.244+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:51.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:51.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:51.828+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:51.829+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:51.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:51.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:51.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:51.767+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:52.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:52.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:52.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:52.276+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:52.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:52.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:52.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:53.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:52.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:52.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:52.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:52.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:52.785+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:53.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:53.315+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:53.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:53.315+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:53.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:53.315+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:53.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:53.316+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:53.755+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:53.755+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:53.755+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:53.756+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:53.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:53.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:53.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:53.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:54.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:54.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:54.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:54.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:54.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:54.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:54.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:54.357+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:55.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:54.633+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41492 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:45:55.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:54.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:55.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:54.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:55.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:54.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:54.802+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:54.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:54.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:54.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:54.778+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:55.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:55.315+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:55.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:55.315+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:55.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:55.315+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:55.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:55.316+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:56.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:55.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:55.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:55.731+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:55.732+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:56.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:55.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:56.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:55.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:56.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:55.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:56.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:55.775+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:56.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:56.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:56.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:56.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:56.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:56.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:56.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:56.289+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:57.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:56.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:57.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:56.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:57.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:56.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:57.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:56.723+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:57.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:56.739+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:57.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:56.739+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:57.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:56.739+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:57.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:56.739+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:57.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:57.253+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:57.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:57.253+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:57.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:57.253+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:57.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:57.254+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:57.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:57.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:57.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:57.705+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:58.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:57.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:58.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:57.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:58.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:57.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:58.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:57.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:58.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:58.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:58.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:58.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:58.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:58.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:58.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:58.282+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:58.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:58.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:58.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:45:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:58.668+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:45:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:58.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:58.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:58.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:45:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:58.758+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:45:59.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:59.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:59.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:59.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:59.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:59.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:45:59.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:45:59.275+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:00.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:45:59.640+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41582 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:59.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:59.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:59.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:45:59.716+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:59.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:59.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:00.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:59.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:00.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:45:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:45:59.741+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:00.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:00.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:46:00.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:00.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:46:00.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:00.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:28.663681+0000 (oldest deadline 2024-02-19T19:35:48.663681+0000) 2024-02-19T19:46:00.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:00.280+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:00.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:00.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:00.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:00.680+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:00.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:00.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:00.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:01.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:00.783+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:01.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:01.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:02.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:01.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:02.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:01.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:02.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:01.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:02.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:01.727+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:01.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:01.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:01.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:01.743+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:02.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:02.321+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:02.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:02.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:02.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:02.739+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:02.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:02.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:02.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:02.735+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:03.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:03.305+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:04.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:03.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:04.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:03.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:04.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:03.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:04.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:03.738+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:03.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:03.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:03.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:32.652141+0000 (oldest deadline 2024-02-19T19:35:52.652141+0000) 2024-02-19T19:46:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:03.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:04.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:04.266+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:05.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:04.647+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41672 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:05.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:04.792+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:04.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:04.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:04.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:04.763+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:05.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:05.239+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:06.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:05.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:06.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:05.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:06.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:05.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:06.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:05.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:35:33.623789+0000 (oldest deadline 2024-02-19T19:35:53.623789+0000) 2024-02-19T19:46:06.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:05.794+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:06.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:06.267+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:07.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:06.812+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:07.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:06.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:07.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:07.277+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:07.771+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:08.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:07.742+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:08.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:08.246+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:09.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:08.788+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:09.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:08.780+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:09.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:09.290+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:09.773+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:09.826+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:10.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:09.654+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41763 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:10.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:10.306+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:11.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:10.751+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:11.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:10.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:11.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:11.295+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:12.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:11.770+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:12.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:11.756+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:12.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:12.321+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:13.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:12.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:13.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:12.730+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:13.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:13.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:14.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:13.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:14.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:13.800+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:14.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:14.342+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:15.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:14.662+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41853 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:15.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:14.749+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:15.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:14.755+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:15.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:15.374+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:16.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:15.738+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:16.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:15.714+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:16.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:16.386+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:17.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:16.719+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:17.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:16.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:17.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:17.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:17.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:17.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:18.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:17.697+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:18.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:18.412+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:19.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:18.715+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:19.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:18.705+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:19.749 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:19.669+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 41942 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:19.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:19.459+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:19.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:19.706+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:20.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:19.749+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:20.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:20.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:21.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:20.761+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:21.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:20.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:21.743 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:21.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:21.743 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:21.660+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:22.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:21.742+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:22.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:22.408+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:23.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:22.785+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:23.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:22.697+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:23.708 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:23.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:23.763+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:24.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:23.708+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:24.677 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:24.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:25.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:24.677+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42032 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:24.776+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:24.714+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:25.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:25.409+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:25.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:25.409+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:25.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:25.409+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:25.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:25.410+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:26.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:25.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:26.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:25.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:26.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:26.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:26.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:26.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:26.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:26.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:26.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:26.369+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:27.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:26.784+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:27.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:26.751+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:27.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:27.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:27.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:27.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:27.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:27.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:27.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:27.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:28.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:27.765+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:28.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:27.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:28.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:27.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:27.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:27.710+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:28.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:28.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:28.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:28.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:28.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:28.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:28.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:28.398+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:29.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:28.785+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:29.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:28.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:29.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:28.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:29.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:28.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:29.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:28.691+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:29.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:29.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:29.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:29.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:29.686 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:29.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:29.686 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:29.352+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:30.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:29.684+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42123 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:30.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:29.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:30.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:29.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:30.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:29.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:30.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:29.798+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:30.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:29.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:30.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:29.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:30.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:29.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:29.719+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:30.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:30.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:30.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:30.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:30.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:30.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:30.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:30.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:31.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:30.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:31.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:30.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:31.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:30.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:31.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:30.836+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:31.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:30.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:31.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:30.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:31.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:30.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:31.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:30.697+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:31.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:31.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:31.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:31.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:31.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:31.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:31.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:31.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:32.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:31.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:32.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:31.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:32.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:31.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:32.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:31.826+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:32.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:31.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:31.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:31.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:31.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:32.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:32.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:32.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:32.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:32.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:32.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:32.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:32.377+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:32.780+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:32.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:32.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:32.782+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:32.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:32.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:32.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:32.636+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:33.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:33.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:33.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:33.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:33.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:33.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:33.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:33.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:33.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:33.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:33.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:33.803+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:34.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:33.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:34.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:33.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:34.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:33.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:34.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:33.668+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:34.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:34.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:34.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:34.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:34.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:34.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:34.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:34.324+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:35.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:34.692+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42214 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:34.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:34.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:34.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:34.798+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:34.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:34.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:34.691+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:34.692+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:35.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:35.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:35.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:35.285+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:35.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:35.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:35.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:35.822+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:35.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:35.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:35.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:35.697+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:36.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:36.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:36.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:36.276+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:36.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:36.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:36.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:36.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:37.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:37.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:37.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:36.746+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:37.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:36.746+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:37.238+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:37.238+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:37.238+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:37.239+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:38.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:37.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:38.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:37.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:38.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:37.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:38.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:37.877+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:37.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:37.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:37.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:37.697+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:38.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:38.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:38.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:38.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:38.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:38.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:38.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:38.257+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:39.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:38.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:39.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:38.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:39.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:38.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:39.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:38.856+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:38.650+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:38.650+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:38.650+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:38.650+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:39.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:39.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:39.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:39.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:39.241+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:39.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:39.699+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42304 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:39.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:39.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:39.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:39.641+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:40.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:39.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:40.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:39.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:40.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:39.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:40.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:39.878+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:40.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:40.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:40.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:40.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:40.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:40.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:40.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:40.285+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:41.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:40.854+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:41.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:40.854+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:41.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:40.854+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:41.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:40.854+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:40.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:40.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:40.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:40.673+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:41.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:41.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:41.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:41.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:41.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:41.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:41.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:41.323+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:42.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:41.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:42.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:41.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:42.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:41.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:42.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:41.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:42.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:41.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:42.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:41.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:41.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:41.630+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:42.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:42.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:42.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:42.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:42.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:42.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:42.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:42.324+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:42.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:42.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:42.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:42.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:42.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:42.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:42.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:43.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:42.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:43.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:43.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:43.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:43.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:43.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:43.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:43.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:43.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:43.865 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:43.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:43.865 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:43.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:43.865 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:43.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:43.865 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:43.587+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:44.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:43.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:44.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:43.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:44.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:43.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:44.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:43.865+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:44.599 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:44.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:44.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:44.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:44.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:44.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:44.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:44.342+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:45.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:44.706+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42394 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:45.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:44.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:45.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:44.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:44.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:44.828+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:44.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:44.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:44.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:44.600+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:45.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:45.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:45.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:45.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:45.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:45.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:45.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:45.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:45.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:45.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:45.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:45.789+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:45.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:45.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:45.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:45.613+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:46.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:46.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:46.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:46.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:46.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:46.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:46.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:46.293+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:46.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:46.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:46.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:46.816+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:46.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:46.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:46.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:46.636+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:47.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:47.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:47.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:47.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:47.263+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:47.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:47.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:47.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:47.826+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:47.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:47.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:47.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:47.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:48.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:48.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:48.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:48.258+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:49.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:48.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:49.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:48.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:49.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:48.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:48.788+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:49.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:48.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:49.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:48.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:49.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:48.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:49.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:48.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:49.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:49.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:49.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:49.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:49.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:49.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:49.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:49.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:49.791 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:49.713+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42485 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:49.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:49.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:49.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:49.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:50.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:49.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:50.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:49.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:50.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:49.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:50.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:49.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:50.280+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:50.280+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:50.280+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:50.281+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:50.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:50.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:50.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:50.479+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:51.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:50.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:51.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:50.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:51.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:50.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:51.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:50.766+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:51.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:51.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:51.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:51.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:51.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:51.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:51.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:51.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:51.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:51.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:51.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:51.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:51.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:51.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:51.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:51.506+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:51.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:51.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:51.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:51.789+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:52.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:52.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:52.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:52.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:52.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:52.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:52.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:52.339+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:52.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:52.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:52.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:52.539+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:53.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:52.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:53.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:52.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:53.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:52.790+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:53.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:52.791+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:53.788 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:53.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:53.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:53.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:53.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:53.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:53.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:53.374+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:53.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:53.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:53.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:53.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:53.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:53.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:53.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:53.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:53.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:53.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:53.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:53.789+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:54.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:54.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:54.721 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:54.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:54.721 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:54.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:54.721 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:54.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:54.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:54.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:54.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:54.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:54.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:54.382+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:55.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:54.720+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42575 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:46:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:54.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:54.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:54.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:54.813+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:55.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:55.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:55.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:55.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:55.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:55.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:55.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:55.396+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:55.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:55.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:55.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:55.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:56.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:55.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:56.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:55.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:56.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:55.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:56.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:55.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:56.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:56.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:56.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:56.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:56.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:56.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:56.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:56.378+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:56.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:56.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:56.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:56.472+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:56.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:56.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:56.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:56.860+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:57.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:57.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:57.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:57.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:57.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:57.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:57.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:57.478+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:57.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:57.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:57.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:57.426+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:57.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:57.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:57.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:57.844+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:58.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:58.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:58.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:58.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:58.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:58.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:58.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:58.426+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:58.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:58.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:58.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:58.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:59.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:58.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:59.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:58.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:59.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:58.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:46:59.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:58.852+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:46:59.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:59.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:59.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:59.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:59.728 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:59.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:46:59.728 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:46:59.446+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:46:59.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:59.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:59.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:59.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:59.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:59.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:46:59.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:46:59.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:00.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:46:59.727+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42665 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:00.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:59.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:00.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:59.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:00.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:59.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:00.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:46:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:46:59.836+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:00.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:00.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:00.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:00.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:00.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:00.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:00.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:00.475+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:00.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:00.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:00.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:00.512+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:01.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:00.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:01.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:00.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:01.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:00.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:01.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:00.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:01.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:01.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:01.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:01.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:01.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:01.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:01.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:01.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:01.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:01.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:01.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:01.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:01.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:02.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:01.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:02.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:01.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:02.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:01.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:02.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:01.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:02.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:02.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:02.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:02.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:02.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:02.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:02.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:02.476+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:02.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:02.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:02.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:02.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:02.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:02.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:02.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:02.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:03.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:03.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:03.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:03.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:03.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:03.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:03.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:03.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:03.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:03.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:03.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:03.492+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:03.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:03.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:03.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:03.838+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:04.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:04.734+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42755 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:04.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:04.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:04.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:04.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:04.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:04.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:04.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:04.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:04.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:04.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:04.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:04.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:04.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:04.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:04.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:04.494+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:05.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:04.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:05.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:04.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:05.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:04.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:05.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:04.867+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:05.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:05.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:05.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:05.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:05.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:05.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:05.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:05.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:05.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:05.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:05.529+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:05.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:05.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:05.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:05.856+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:06.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:06.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:06.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:06.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:06.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:06.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:06.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:06.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:06.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:06.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:06.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:06.476+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:06.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:06.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:06.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:06.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:07.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:07.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:07.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:07.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:07.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:07.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:07.788 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:07.507+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:07.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:07.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:07.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:07.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:07.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:07.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:07.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:07.477+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:07.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:07.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:07.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:07.787+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:08.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:08.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:08.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:08.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:08.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:08.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:08.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:08.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:08.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:08.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:08.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:08.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:08.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:08.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:08.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:08.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:09.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:08.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:09.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:08.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:09.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:08.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:09.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:08.758+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:09.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:09.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:09.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:09.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:09.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:09.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:09.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:09.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:09.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:09.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:09.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:09.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:09.743 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:09.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:09.743 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:09.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:10.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:09.741+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42845 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:09.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:09.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:09.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:09.776+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:10.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:10.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:10.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:10.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:10.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:10.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:10.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:10.461+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:10.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:10.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:10.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:10.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:10.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:10.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:10.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:10.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:11.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:11.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:11.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:11.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:11.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:11.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:11.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:11.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:11.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:11.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:11.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:12.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:11.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:11.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:11.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:11.838+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:12.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:12.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:12.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:12.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:12.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:12.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:12.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:12.380+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:12.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:12.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:12.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:12.538+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:12.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:12.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:12.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:12.846+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:13.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:13.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:13.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:13.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:13.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:13.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:13.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:13.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:13.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:13.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:13.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:13.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:13.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:13.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:13.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:13.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:13.808+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:14.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:14.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:14.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:14.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:14.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:14.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:14.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:14.386+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:14.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:14.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:14.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:14.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:14.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:14.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:14.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:14.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:15.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:14.748+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 42936 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:15.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:14.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:15.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:14.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:15.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:14.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:15.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:14.763+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:15.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:15.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:15.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:15.532+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:15.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:15.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:15.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:15.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:15.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:15.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:15.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:15.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:15.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:15.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:15.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:15.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:16.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:16.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:16.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:16.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:16.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:16.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:16.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:16.542+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:16.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:16.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:16.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:16.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:16.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:16.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:16.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:16.386+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:17.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:16.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:17.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:16.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:17.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:16.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:17.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:16.767+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:17.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:17.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:17.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:17.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:17.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:17.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:17.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:17.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:17.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:17.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:17.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:17.539+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:18.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:17.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:18.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:17.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:18.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:17.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:18.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:17.810+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:18.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:18.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:18.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:18.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:18.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:18.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:18.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:18.396+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:18.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:18.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:18.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:18.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:18.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:18.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:18.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:18.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:18.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:18.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:18.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:18.850+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:19.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:19.384+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:19.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:19.384+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:19.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:19.384+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:19.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:19.385+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:19.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:19.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:19.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:19.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:19.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:19.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:19.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:19.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:20.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:19.755+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43025 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:20.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:19.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:20.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:19.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:20.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:19.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:20.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:19.835+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:20.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:20.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:20.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:20.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:20.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:20.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:20.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:20.375+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:20.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:20.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:20.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:20.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:20.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:20.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:20.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:20.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:21.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:20.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:21.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:20.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:21.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:20.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:21.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:20.790+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:21.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:21.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:21.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:21.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:21.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:21.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:21.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:21.356+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:21.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:21.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:21.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:21.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:21.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:21.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:21.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:21.525+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:22.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:21.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:22.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:21.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:22.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:21.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:22.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:21.759+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:22.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:22.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:22.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:22.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:22.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:22.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:22.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:22.329+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:22.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:22.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:22.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:22.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:22.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:22.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:22.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:22.513+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:23.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:22.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:23.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:22.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:23.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:22.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:23.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:22.740+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:23.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:23.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:23.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:23.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:23.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:23.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:23.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:23.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:23.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:23.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:23.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:23.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:23.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:23.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:23.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:23.477+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:23.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:23.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:23.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:23.775+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:24.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:24.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:24.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:24.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:24.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:24.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:24.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:24.506+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:24.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:24.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:24.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:24.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:24.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:24.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:24.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:24.333+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:25.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:24.763+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43116 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:24.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:24.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:24.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:24.770+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:25.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:25.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:25.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:25.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:25.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:25.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:25.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:25.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:25.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:25.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:25.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:25.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:25.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:25.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:25.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:25.302+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:26.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:25.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:26.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:25.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:26.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:25.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:26.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:25.813+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:26.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:26.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:26.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:26.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:26.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:26.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:26.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:26.290+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:26.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:26.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:26.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:26.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:26.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:26.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:26.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:26.500+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:27.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:26.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:27.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:26.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:27.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:26.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:27.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:26.789+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:27.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:27.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:27.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:27.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:27.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:27.287+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:27.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:27.288+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:27.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:27.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:27.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:27.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:27.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:28.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:27.804+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:28.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:27.804+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:28.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:27.804+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:28.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:27.805+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:28.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:28.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:28.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:28.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:28.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:28.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:28.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:28.282+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:28.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:28.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:28.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:28.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:28.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:29.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:28.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:29.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:28.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:29.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:28.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:29.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:28.822+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:29.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:29.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:29.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:29.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:29.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:29.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:29.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:29.310+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:29.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:29.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:29.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:29.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:29.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:29.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:29.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:29.424+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:30.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:29.768+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43206 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:30.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:29.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:30.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:29.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:30.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:29.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:30.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:29.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:30.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:30.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:30.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:30.285+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:30.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:30.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:30.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:30.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:31.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:30.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:31.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:30.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:31.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:30.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:31.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:30.869+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:31.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:31.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:31.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:31.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:31.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:31.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:31.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:31.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:31.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:31.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:31.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:31.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:31.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:32.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:31.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:32.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:31.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:32.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:31.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:32.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:31.824+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:32.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:32.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:32.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:32.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:32.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:32.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:32.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:32.302+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:32.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:32.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:32.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:32.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:32.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:32.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:32.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:32.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:32.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:32.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:32.776+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:33.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:32.777+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:33.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:33.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:33.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:33.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:33.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:33.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:33.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:33.421+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:33.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:33.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:33.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:33.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:33.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:33.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:33.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:33.307+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:33.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:33.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:33.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:33.731+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:34.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:34.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:34.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:34.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:34.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:34.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:34.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:34.462+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:34.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:34.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:34.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:34.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:34.277+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:35.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:34.774+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43297 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:34.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:34.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:34.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:34.686+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:35.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:35.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:35.259+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:35.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:35.260+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:35.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:35.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:35.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:35.475+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:36.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:35.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:35.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:35.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:35.732+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:36.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:36.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:36.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:36.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:36.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:36.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:36.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:36.219+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:36.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:36.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:36.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:36.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:36.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:36.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:36.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:36.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:37.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:36.782+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:37.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:37.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:37.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:37.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:37.240+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:37.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:37.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:37.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:37.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:37.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:37.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:38.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:37.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:38.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:37.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:38.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:37.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:38.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:37.773+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:38.450 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:38.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:38.450 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:38.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:38.450 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:38.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:38.450 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:38.225+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:38.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:38.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:38.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:38.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:38.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:38.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:38.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:38.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:39.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:38.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:39.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:38.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:39.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:38.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:39.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:38.749+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:39.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:39.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:39.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:39.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:39.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:39.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:39.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:39.263+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:39.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:39.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:39.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:39.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:39.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:39.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:39.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:39.494+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:40.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:39.779+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43386 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:40.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:39.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:40.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:39.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:40.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:39.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:40.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:39.750+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:40.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:40.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:40.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:40.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:40.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:40.303+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:40.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:40.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:40.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:40.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:40.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:40.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:40.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:40.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:40.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:40.526+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:40.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:40.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:40.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:40.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:40.705 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:40.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:41.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:40.704+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:41.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:41.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:41.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:41.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:41.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:41.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:41.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:41.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:41.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:41.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:41.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:41.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:41.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:41.299+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:41.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:41.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:42.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:41.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:42.074 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:41.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:42.074 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:41.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:42.075 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:41.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:42.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:42.250+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:42.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:42.250+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:42.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:42.250+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:42.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:42.251+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:42.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:42.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:42.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:42.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:42.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:42.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:43.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:42.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:42.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:42.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:42.668+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:43.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:43.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:43.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:43.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:43.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:43.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:43.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:43.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:43.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:43.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:43.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:43.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:43.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:43.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:43.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:43.512+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:43.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:43.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:43.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:43.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:43.682+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:44.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:44.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:44.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:44.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:44.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:44.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:44.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:44.200+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:44.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:44.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:44.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:44.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:44.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:44.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:44.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:44.531+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:44.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:44.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:44.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:44.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:44.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:44.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:44.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:44.664+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:45.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:44.785+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43479 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:45.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:45.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:45.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:45.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:45.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:45.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:45.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:45.161+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:45.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:45.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:45.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:45.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:45.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:45.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:45.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:45.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:45.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:45.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:45.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:45.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:45.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:45.649+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:46.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:46.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:46.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:46.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:46.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:46.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:46.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:46.157+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:46.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:46.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:46.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:46.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:46.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:46.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:46.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:46.669+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:46.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:46.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:46.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:46.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:46.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:46.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:46.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:47.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:47.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:47.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:47.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:47.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:47.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:47.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:47.202+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:47.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:47.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:47.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:47.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:47.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:47.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:47.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:47.676+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:47.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:47.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:47.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:47.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:47.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:47.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:47.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:47.489+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:48.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:48.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:48.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:48.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:48.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:48.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:48.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:48.153+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:48.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:48.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:48.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:48.664+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:48.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:48.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:48.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:48.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:48.521+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:49.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:49.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:49.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:49.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:49.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:49.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:49.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:49.179+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:49.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:49.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:49.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:49.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:49.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:49.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:49.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:49.693+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:50.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:49.791+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43569 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:50.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:50.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:50.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:50.153+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:50.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:50.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:50.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:50.584+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:51.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:50.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:50.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:50.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:50.648+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:51.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:51.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:51.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:51.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:51.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:51.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:51.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:51.114+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:51.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:51.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:51.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:51.640+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:51.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:52.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:51.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:52.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:51.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:52.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:51.604+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:52.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:52.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:52.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:52.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:52.123+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:52.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:52.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:52.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:52.610+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:53.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:52.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:52.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:52.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:52.632+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:53.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:53.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:53.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:53.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:53.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:53.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:53.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:53.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:53.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:53.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:53.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:53.575+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:53.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:54.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:53.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:53.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:53.628+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:54.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:54.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:54.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:54.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:54.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:54.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:54.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:54.112+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:54.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:54.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:54.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:54.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:54.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:54.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:54.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:54.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:54.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:54.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:54.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:54.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:54.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:54.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:54.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:54.628+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:55.136 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:54.797+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43660 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:47:55.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:55.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:55.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:55.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:55.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:55.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:55.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:55.137+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:56.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:55.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:56.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:55.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:56.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:55.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:56.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:55.674+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:56.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:55.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:56.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:55.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:56.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:55.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:56.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:55.548+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:56.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:56.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:56.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:56.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:56.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:56.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:56.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:56.184+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:56.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:56.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:56.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:56.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:56.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:56.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:56.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:56.692+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:56.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:56.500+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:56.500+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:56.500+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:56.501+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:57.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:57.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:57.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:57.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:57.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:57.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:57.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:57.188+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:57.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:57.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:57.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:57.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:57.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:57.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:57.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:57.476+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:57.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:57.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:57.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:57.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:57.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:57.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:57.683+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:58.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:58.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:58.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:58.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:58.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:58.186+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:58.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:58.187+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:58.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:58.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:58.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:58.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:58.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:58.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:58.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:58.458+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:58.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:58.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:58.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:47:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:58.724+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:47:59.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:59.230+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:59.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:59.230+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:59.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:59.230+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:47:59.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:47:59.231+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:47:59.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:59.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:59.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:59.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:59.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:59.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:47:59.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:47:59.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:00.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:47:59.802+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43749 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:00.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:59.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:00.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:59.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:00.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:59.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:00.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:47:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:47:59.756+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:00.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:00.213+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:00.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:00.213+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:00.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:00.213+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:00.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:00.214+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:00.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:00.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:00.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:00.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:00.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:00.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:00.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:00.715+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:00.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:00.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:00.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:00.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:00.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:00.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:00.498+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:01.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:01.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:01.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:01.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:01.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:01.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:01.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:01.232+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:01.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:01.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:01.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:01.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:01.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:01.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:01.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:01.672+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:01.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:01.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:01.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:01.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:01.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:02.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:02.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:02.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:02.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:02.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:02.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:02.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:02.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:02.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:02.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:02.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:02.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:02.275+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:03.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:02.718+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:02.718+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:02.718+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:02.719+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:03.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:03.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:03.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:03.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:03.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:03.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:03.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:03.293+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:03.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:03.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:03.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:03.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:03.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:03.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:03.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:03.464+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:04.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:03.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:04.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:03.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:04.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:03.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:04.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:03.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:04.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:04.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:04.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:04.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:04.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:04.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:04.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:04.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:04.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:04.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:04.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:04.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:04.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:04.266+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:04.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:04.267+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:05.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:04.807+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43834 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:05.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:04.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:05.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:04.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:05.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:04.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:05.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:04.757+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:05.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:05.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:05.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:05.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:05.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:05.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:05.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:05.277+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:05.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:05.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:05.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:05.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:05.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:05.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:05.477+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:06.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:05.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:06.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:05.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:06.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:05.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:06.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:05.783+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:06.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:06.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:06.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:06.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:06.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:06.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:06.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:06.275+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:06.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:06.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:06.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:06.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:06.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:06.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:06.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:06.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:07.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:06.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:07.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:06.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:07.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:06.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:07.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:06.781+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:07.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:07.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:07.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:07.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:07.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:07.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:07.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:07.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:07.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:07.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:07.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:07.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:07.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:07.270+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:07.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:07.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:07.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:07.755+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:08.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:08.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:08.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:08.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:08.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:08.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:08.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:08.314+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:08.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:08.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:08.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:08.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:08.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:08.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:08.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:08.455+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:09.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:08.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:09.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:08.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:09.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:08.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:09.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:08.763+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:09.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:09.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:09.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:09.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:09.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:09.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:09.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:09.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:09.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:09.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:09.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:09.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:09.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:09.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:09.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:09.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:10.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:09.813+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43915 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:09.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:09.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:09.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:09.750+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:10.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:10.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:10.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:10.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:10.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:10.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:10.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:10.327+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:10.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:10.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:10.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:10.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:10.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:10.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:10.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:10.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:11.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:10.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:11.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:10.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:11.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:10.781+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:11.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:10.782+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:11.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:11.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:11.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:11.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:11.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:11.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:11.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:11.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:11.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:11.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:11.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:11.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:11.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:11.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:11.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:11.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:11.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:12.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:12.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:12.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:12.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:12.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:12.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:12.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:12.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:12.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:12.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:12.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:12.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:12.293+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:13.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:12.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:13.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:12.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:13.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:12.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:13.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:12.872+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:13.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:13.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:13.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:13.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:13.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:13.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:13.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:13.289+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:13.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:13.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:13.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:13.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:13.418+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:13.419+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:14.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:13.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:14.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:13.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:14.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:13.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:14.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:13.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:14.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:14.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:14.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:14.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:14.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:14.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:14.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:14.256+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:14.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:14.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:14.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:14.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:14.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:14.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:14.390+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:15.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:14.819+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 43997 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:15.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:14.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:15.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:14.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:15.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:14.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:15.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:14.812+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:15.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:15.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:15.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:15.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:15.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:15.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:15.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:15.290+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:15.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:15.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:15.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:15.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:15.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:15.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:16.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:15.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:15.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:15.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:15.816+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:16.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:16.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:16.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:16.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:16.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:16.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:16.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:16.326+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:16.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:16.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:16.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:16.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:16.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:16.396+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:17.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:16.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:17.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:16.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:17.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:16.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:17.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:16.799+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:17.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:17.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:17.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:17.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:17.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:17.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:17.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:17.371+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:17.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:17.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:17.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:17.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:17.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:17.291+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:18.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:17.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:18.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:17.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:18.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:17.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:18.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:17.843+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:18.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:18.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:18.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:18.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:18.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:18.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:18.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:18.244+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:18.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:18.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:18.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:18.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:18.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:18.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:18.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:18.347+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:18.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:18.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:18.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:18.833+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:19.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:19.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:19.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:19.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:19.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:19.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:19.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:19.292+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:19.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:19.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:19.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:19.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:19.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:20.285 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:19.824+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44077 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:20.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:19.798+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:20.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:19.798+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:20.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:19.798+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:20.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:19.798+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:20.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:20.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:20.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:20.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:20.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:20.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:20.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:20.285+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:20.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:20.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:20.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:20.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:20.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:20.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:20.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:20.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:21.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:20.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:21.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:20.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:21.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:20.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:21.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:20.809+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:21.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:21.251+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:21.251+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:21.251+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:21.252+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:21.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:21.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:21.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:21.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:21.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:21.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:21.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:21.350+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:22.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:21.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:22.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:21.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:22.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:21.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:22.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:21.797+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:22.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:22.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:22.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:22.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:22.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:22.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:22.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:22.284+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:22.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:22.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:22.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:22.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:22.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:22.397+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:23.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:22.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:23.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:22.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:23.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:22.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:23.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:22.788+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:23.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:23.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:23.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:23.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:23.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:23.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:23.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:23.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:23.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:23.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:23.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:23.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:23.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:23.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:23.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:23.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:24.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:23.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:23.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:23.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:23.744+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:24.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:24.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:24.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:24.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:24.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:24.284+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:24.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:24.284+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:24.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:24.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:24.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:24.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:25.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:24.830+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44159 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:24.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:24.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:24.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:24.767+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:25.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:25.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:25.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:25.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:25.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:25.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:25.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:25.277+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:25.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:25.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:25.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:25.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:25.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:25.375+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:26.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:25.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:26.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:25.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:26.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:25.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:26.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:25.720+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:26.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:26.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:26.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:26.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:26.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:26.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:26.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:26.235+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:26.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:26.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:26.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:26.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:26.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:26.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:26.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:26.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:27.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:26.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:27.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:26.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:27.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:26.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:27.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:26.742+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:27.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:27.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:27.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:27.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:27.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:27.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:27.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:27.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:27.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:27.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:27.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:27.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:27.247+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:28.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:27.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:28.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:27.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:28.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:27.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:28.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:27.724+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:28.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:28.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:28.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:28.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:28.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:28.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:28.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:28.230+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:28.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:28.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:28.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:28.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:28.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:28.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:28.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:29.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:28.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:29.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:28.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:29.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:28.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:29.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:28.764+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:29.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:29.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:29.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:29.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:29.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:29.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:29.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:29.232+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:29.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:29.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:29.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:29.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:29.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:29.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:29.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:29.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:29.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:29.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:29.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:29.726+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:30.192 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:29.837+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44246 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:30.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:30.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:30.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:30.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:30.193+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:30.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:30.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:30.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:30.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:30.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:31.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:30.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:31.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:30.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:31.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:30.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:31.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:30.689+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:31.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:31.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:31.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:31.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:31.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:31.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:31.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:31.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:31.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:31.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:31.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:31.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:31.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:31.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:31.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:31.668+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:31.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:31.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:31.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:31.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:31.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:31.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:31.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:32.462 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:32.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:32.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:32.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:32.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:32.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:32.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:32.129+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:32.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:32.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:32.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:32.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:32.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:32.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:32.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:32.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:32.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:32.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:32.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:32.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:32.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:32.704+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:33.414 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:33.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:33.414 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:33.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:33.414 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:33.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:33.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:33.102+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:33.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:33.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:33.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:33.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:33.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:33.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:33.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:33.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:34.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:33.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:34.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:33.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:33.753+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:34.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:33.754+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:34.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:34.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:34.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:34.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:34.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:34.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:34.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:34.122+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:34.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:34.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:34.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:34.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:34.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:34.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:34.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:34.408+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:35.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:34.842+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44337 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:34.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:34.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:34.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:35.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:34.780+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:35.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:35.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:35.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:35.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:35.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:35.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:35.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:35.120+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:35.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:35.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:35.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:35.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:35.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:35.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:35.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:35.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:36.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:35.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:35.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:35.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:36.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:35.737+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:36.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:36.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:36.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:36.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:36.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:36.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:36.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:36.075+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:36.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:36.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:36.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:36.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:36.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:36.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:36.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:36.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:37.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:36.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:37.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:36.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:36.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:36.692+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:37.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:37.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:37.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:37.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:37.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:37.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:37.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:37.088+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:37.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:37.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:37.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:37.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:37.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:37.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:37.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:37.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:37.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:37.678+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:37.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:37.678+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:37.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:37.678+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:37.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:37.678+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:38.447 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:38.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:38.447 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:38.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:38.447 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:38.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:38.447 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:38.088+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:38.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:38.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:38.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:38.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:38.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:38.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:38.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:38.640+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:38.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:38.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:38.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:38.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:38.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:38.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:38.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:38.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:39.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:39.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:39.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:39.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:39.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:39.118+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:39.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:39.118+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:39.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:39.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:39.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:39.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:39.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:39.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:39.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:39.643+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:39.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:39.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:39.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:39.452+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:40.125 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:39.848+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44426 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:40.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:40.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:40.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:40.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:40.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:40.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:40.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:40.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:40.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:40.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:40.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:40.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:40.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:40.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:40.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:40.639+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:40.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:40.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:40.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:40.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:40.492+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:41.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:41.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:41.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:41.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:41.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:41.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:41.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:41.147+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:41.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:41.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:41.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:41.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:41.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:41.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:41.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:41.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:41.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:41.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:41.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:41.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:41.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:41.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:41.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:41.681+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:42.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:42.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:42.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:42.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:42.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:42.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:42.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:42.150+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:42.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:42.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:42.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:42.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:42.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:42.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:42.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:42.671+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:42.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:42.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:42.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:42.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:42.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:42.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:42.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:42.518+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:43.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:43.172+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:43.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:43.172+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:43.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:43.172+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:43.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:43.172+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:43.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:43.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:43.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:43.658+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:43.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:43.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:43.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:43.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:43.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:43.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:43.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:43.470+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:44.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:44.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:44.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:44.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:44.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:44.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:44.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:44.145+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:44.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:44.500+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:44.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:44.500+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:44.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:44.500+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:44.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:44.501+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:44.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:44.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:44.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:44.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:44.691+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:45.159 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:44.854+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44517 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:45.467 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:45.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:45.468 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:45.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:45.468 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:45.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:45.468 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:45.159+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:45.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:45.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:45.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:45.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:45.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:45.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:45.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:45.468+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:45.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:45.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:45.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:45.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:45.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:45.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:46.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:46.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:46.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:46.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:46.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:46.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:46.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:46.182+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:46.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:46.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:46.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:46.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:46.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:46.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:46.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:46.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:46.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:46.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:46.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:46.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:46.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:46.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:46.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:46.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:47.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:47.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:47.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:47.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:47.232+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:47.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:47.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:47.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:47.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:47.478+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:48.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:47.637+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:47.637+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:47.637+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:47.638+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:48.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:48.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:48.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:48.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:48.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:48.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:48.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:48.201+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:48.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:48.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:48.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:48.635+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:48.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:48.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:48.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:48.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:48.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:49.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:49.171+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:49.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:49.171+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:49.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:49.171+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:49.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:49.171+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:49.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:49.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:49.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:49.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:49.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:49.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:49.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:49.590+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:49.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:49.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:49.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:49.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:50.158 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:49.859+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44607 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:50.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:50.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:50.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:50.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:50.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:50.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:50.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:50.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:50.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:50.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:50.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:50.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:50.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:50.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:50.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:50.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:50.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:50.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:50.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:50.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:50.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:50.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:50.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:50.556+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:51.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:51.189+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:51.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:51.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:51.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:51.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:51.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:51.190+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:51.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:51.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:51.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:51.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:51.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:51.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:51.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:51.514+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:51.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:51.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:51.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:51.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:51.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:52.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:52.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:52.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:52.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:52.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:52.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:52.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:52.200+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:52.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:52.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:52.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:52.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:52.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:52.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:52.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:52.490+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:52.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:52.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:52.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:52.508+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:53.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:53.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:53.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:53.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:53.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:53.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:53.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:53.160+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:53.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:53.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:53.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:53.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:53.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:53.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:53.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:53.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:53.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:53.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:53.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:53.516+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:54.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:54.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:54.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:54.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:54.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:54.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:54.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:54.164+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:54.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:54.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:54.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:54.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:54.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:54.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:54.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:54.522+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:54.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:54.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:54.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:54.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:55.143 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:54.865+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44698 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:48:55.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:55.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:55.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:55.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:55.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:55.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:55.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:55.144+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:55.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:55.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:55.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:55.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:55.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:55.543+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:55.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:55.543+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:55.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:55.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:55.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:55.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:55.435+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:56.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:56.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:56.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:56.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:56.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:56.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:56.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:56.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:56.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:56.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:56.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:56.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:56.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:56.150+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:56.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:56.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:56.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:56.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:56.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:56.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:56.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:56.396+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:56.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:56.537+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:57.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:57.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:57.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:57.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:57.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:57.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:57.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:57.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:57.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:57.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:57.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:57.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:57.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:57.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:57.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:57.420+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:57.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:57.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:57.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:57.557+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:58.428 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:58.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:58.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:58.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:58.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:58.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:58.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:58.130+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:58.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:58.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:58.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:58.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:58.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:58.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:58.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:58.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:58.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:58.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:58.599+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:58.600+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:59.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:59.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:59.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:59.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:59.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:59.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:48:59.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:48:59.087+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:48:59.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:59.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:59.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:59.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:59.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:59.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:48:59.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:48:59.622+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:48:59.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:59.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:59.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:59.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:48:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:48:59.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:00.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:48:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:48:59.871+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44787 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:00.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:00.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:00.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:00.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:00.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:00.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:00.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:00.086+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:00.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:00.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:00.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:00.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:00.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:00.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:00.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:00.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:00.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:00.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:00.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:00.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:00.661+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:01.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:01.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:01.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:01.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:01.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:01.123+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:01.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:01.124+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:01.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:01.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:01.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:01.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:01.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:01.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:01.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:01.627+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:01.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:01.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:01.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:01.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:01.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:02.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:02.078+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:02.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:02.078+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:02.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:02.078+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:02.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:02.079+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:02.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:02.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:02.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:02.619+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:02.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:02.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:02.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:02.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:02.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:02.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:03.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:03.043+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:03.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:03.043+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:03.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:03.043+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:03.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:03.044+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:03.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:03.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:03.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:03.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:03.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:03.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:03.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:03.582+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:03.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:03.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:03.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:03.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:03.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:03.466+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:04.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:04.035+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:04.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:04.035+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:04.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:04.035+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:04.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:04.035+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:04.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:04.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:04.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:04.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:04.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:04.553+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:04.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:04.554+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:04.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:04.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:04.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:04.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:04.496+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:05.290 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:04.877+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44878 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:05.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:05.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:05.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:05.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:05.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:05.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:05.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:05.065+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:05.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:05.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:05.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:05.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:05.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:05.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:05.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:05.524+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:05.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:05.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:05.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:05.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:05.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:05.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:05.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:05.530+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:06.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:06.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:06.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:06.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:06.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:06.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:06.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:06.018+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:06.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:06.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:06.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:06.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:06.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:06.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:06.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:06.492+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:06.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:06.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:06.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:06.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:06.575+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:07.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:07.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:07.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:07.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:07.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:07.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:07.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:07.049+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:07.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:07.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:07.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:07.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:07.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:07.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:07.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:07.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:07.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:07.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:07.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:07.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:07.571+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:08.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:08.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:08.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:08.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:08.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:08.028+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:08.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:08.028+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:08.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:08.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:08.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:08.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:08.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:08.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:08.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:08.434+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:09.020 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:08.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:09.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:08.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:09.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:08.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:09.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:08.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:09.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:09.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:09.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:09.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:09.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:09.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:09.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:09.021+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:09.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:09.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:09.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:09.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:09.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:09.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:09.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:09.479+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:09.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:09.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:09.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:09.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:09.547+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:10.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:09.883+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 44968 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:10.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:10.055+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:10.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:10.055+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:10.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:10.055+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:10.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:10.055+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:10.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:10.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:10.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:10.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:10.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:10.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:10.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:10.513+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:10.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:10.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:10.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:11.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:11.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:11.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:11.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:11.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:11.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:11.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:11.069+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:11.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:11.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:11.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:11.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:11.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:11.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:11.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:11.559+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:11.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:11.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:11.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:11.510+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:12.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:12.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:12.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:12.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:12.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:12.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:12.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:12.029+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:12.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:12.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:12.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:12.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:12.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:12.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:12.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:12.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:12.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:12.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:12.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:12.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:12.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:12.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:13.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:12.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:13.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:12.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:13.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:12.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:13.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:12.988+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:13.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:13.552+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:13.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:13.552+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:13.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:13.552+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:13.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:13.553+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:13.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:13.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:13.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:13.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:13.479+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:14.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:13.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:14.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:13.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:14.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:13.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:14.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:13.939+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:14.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:14.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:14.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:14.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:14.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:14.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:14.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:14.511+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:14.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:14.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:14.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:14.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:14.464+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:15.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:14.889+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45059 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:15.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:14.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:15.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:14.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:15.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:14.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:15.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:14.962+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:15.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:15.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:15.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:15.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:15.492+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:15.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:15.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:15.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:15.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:15.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:15.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:15.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:15.514+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:16.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:15.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:16.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:15.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:16.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:15.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:16.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:15.955+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:16.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:16.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:16.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:16.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:16.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:16.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:16.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:16.504+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:16.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:16.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:16.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:16.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:16.525+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:17.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:16.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:17.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:16.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:17.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:16.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:17.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:16.935+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:17.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:17.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:17.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:17.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:17.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:17.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:17.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:17.504+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:17.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:17.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:17.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:17.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:17.473+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:18.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:17.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:18.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:17.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:18.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:17.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:18.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:17.907+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:18.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:18.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:18.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:18.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:18.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:18.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:18.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:18.501+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:18.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:18.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:18.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:18.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:18.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:18.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:18.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:18.443+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:19.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:18.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:19.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:18.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:19.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:18.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:19.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:18.953+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:19.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:19.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:19.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:19.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:19.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:19.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:19.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:19.409+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:19.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:19.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:19.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:19.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:19.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:20.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:19.894+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45148 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:20.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:19.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:20.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:19.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:20.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:19.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:20.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:19.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:20.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:20.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:20.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:20.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:20.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:20.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:20.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:20.455+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:20.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:20.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:20.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:20.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:20.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:21.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:20.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:21.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:20.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:21.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:20.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:21.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:20.993+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:21.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:21.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:21.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:21.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:21.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:21.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:21.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:21.494+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:21.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:21.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:21.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:21.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:21.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:22.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:21.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:22.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:21.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:22.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:21.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:22.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:22.000+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:22.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:22.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:22.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:22.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:22.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:22.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:22.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:22.384+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:22.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:22.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:22.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:22.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:22.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:22.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:22.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:22.484+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:23.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:23.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:23.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:23.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:23.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:23.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:23.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:23.027+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:23.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:23.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:23.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:23.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:23.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:23.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:23.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:23.342+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:23.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:23.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:23.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:23.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:23.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:23.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:23.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:23.493+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:24.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:24.006+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:24.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:24.006+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:24.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:24.006+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:24.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:24.007+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:24.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:24.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:24.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:24.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:24.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:24.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:24.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:24.486+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:24.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:24.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:24.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:24.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:24.295+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:25.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:24.900+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45239 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:25.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:25.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:25.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:25.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:25.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:25.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:25.045+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:25.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:25.444+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:25.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:25.444+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:25.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:25.444+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:25.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:25.445+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:25.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:25.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:25.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:25.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:25.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:25.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:25.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:25.300+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:26.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:26.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:26.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:26.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:26.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:26.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:26.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:26.031+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:26.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:26.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:26.254+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:26.255+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:26.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:26.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:26.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:26.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:26.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:26.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:26.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:26.413+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:27.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:26.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:27.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:26.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:27.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:26.985+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:27.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:26.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:27.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:27.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:27.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:27.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:27.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:27.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:27.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:27.434+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:27.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:27.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:27.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:27.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:28.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:27.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:28.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:27.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:28.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:27.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:28.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:27.945+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:28.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:28.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:28.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:28.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:28.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:28.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:28.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:28.426+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:28.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:28.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:28.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:28.315+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:29.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:28.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:29.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:28.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:29.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:28.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:29.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:28.996+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:29.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:29.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:29.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:29.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:29.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:29.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:29.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:29.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:29.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:30.287 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:29.906+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45289 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:30.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:30.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:30.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:30.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:30.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:30.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:30.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:30.024+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:30.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:30.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:30.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:30.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:30.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:30.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:30.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:30.287+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:30.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:30.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:30.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:30.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:30.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:30.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:30.974+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:30.975+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:31.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:31.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:31.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:31.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:31.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:31.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:31.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:31.321+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:31.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:31.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:31.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:31.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:31.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:31.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:31.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:31.400+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:32.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:31.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:32.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:31.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:32.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:31.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:32.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:31.978+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:32.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:32.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:32.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:32.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:32.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:32.294+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:32.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:32.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:32.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:32.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:32.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:32.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:32.404+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:33.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:33.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:33.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:33.020+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:33.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:33.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:33.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:33.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:33.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:33.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:33.449+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:33.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:33.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:33.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:33.309+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:33.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:33.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:33.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:33.972+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:34.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:34.421+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:34.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:34.421+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:34.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:34.421+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:34.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:34.422+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:34.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:34.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:34.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:34.295+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:35.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:34.912+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45335 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:35.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:35.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:35.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:35.006+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:35.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:35.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:35.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:35.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:35.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:35.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:35.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:35.428+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:35.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:35.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:35.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:35.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:35.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:35.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:35.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:35.301+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:36.261 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:35.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:36.262 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:35.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:36.262 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:35.993+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:36.262 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:35.994+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:36.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:36.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:36.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:36.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:36.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:36.461+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:36.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:36.461+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:36.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:36.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:36.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:36.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:36.262+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:37.248 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:36.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:37.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:36.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:37.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:36.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:37.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:36.996+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:37.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:37.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:37.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:37.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:37.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:37.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:37.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:37.443+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:37.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:37.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:37.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:37.248+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:38.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:38.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:38.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:38.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:38.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:38.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:38.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:38.002+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:38.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:38.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:38.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:38.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:38.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:38.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:38.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:38.488+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:38.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:38.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:38.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:38.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:38.277+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:39.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:38.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:39.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:38.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:39.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:38.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:39.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:38.973+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:39.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:39.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:39.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:39.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:39.315+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:39.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:39.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:39.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:39.502+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:40.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:39.918+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45379 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:40.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:40.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:40.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:40.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:40.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:40.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:40.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:40.001+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:40.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:40.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:40.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:40.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:40.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:40.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:40.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:40.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:40.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:40.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:40.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:40.545+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:41.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:41.050+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:41.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:41.050+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:41.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:41.050+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:41.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:41.050+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:41.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:41.577+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:41.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:41.577+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:41.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:41.577+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:41.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:41.578+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:41.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:41.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:41.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:41.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:41.385+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:42.336 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:42.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:42.337 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:42.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:42.337 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:42.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:42.337 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:42.052+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:42.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:42.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:42.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:42.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:42.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:42.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:42.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:42.336+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:42.616+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:42.616+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:42.616+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:42.617+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:43.343 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:43.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:43.343 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:43.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:43.343 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:43.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:43.343 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:43.087+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:43.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:43.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:43.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:43.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:43.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:43.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:43.654 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:43.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:44.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:43.652+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:44.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:43.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:44.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:43.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:44.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:43.654+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:44.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:44.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:44.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:44.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:44.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:44.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:44.373 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:44.122+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:44.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:44.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:44.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:44.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:44.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:44.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:44.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:44.371+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:45.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:44.924+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45425 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:45.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:44.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:45.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:44.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:45.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:44.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:45.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:44.704+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:45.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:45.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:45.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:45.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:45.373 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:45.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:45.373 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:45.139+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:45.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:45.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:45.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:45.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:45.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:45.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:45.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:45.372+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:45.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:45.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:45.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:45.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:46.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:46.334+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:46.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:46.334+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:46.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:46.334+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:46.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:46.334+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:46.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:46.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:46.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:46.115+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:46.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:46.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:46.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:46.772+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:47.370 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:47.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:47.371 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:47.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:47.371 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:47.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:47.371 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:47.074+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:47.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:47.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:47.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:47.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:47.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:48.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:47.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:48.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:47.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:48.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:47.815+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:48.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:47.816+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:48.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:48.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:48.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:48.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:48.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:48.092+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:48.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:48.093+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:48.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:48.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:48.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:48.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:48.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:48.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:48.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:48.406+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:49.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:48.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:49.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:48.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:49.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:48.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:49.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:48.818+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:49.357 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:49.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:49.357 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:49.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:49.357 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:49.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:49.357 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:49.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:49.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:49.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:49.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:49.357+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:50.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:49.930+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45470 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:50.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:49.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:50.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:49.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:49.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:49.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:50.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:50.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:50.054+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:50.055+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:50.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:50.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:50.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:50.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:50.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:50.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:50.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:50.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:51.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:50.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:51.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:50.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:51.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:50.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:51.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:50.802+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:51.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:51.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:51.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:51.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:51.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:51.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:51.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:51.081+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:51.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:51.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:51.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:51.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:51.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:51.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:51.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:51.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:51.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:51.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:52.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:51.790+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:52.389 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:52.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:52.389 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:52.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:52.389 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:52.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:52.389 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:52.132+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:52.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:52.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:52.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:52.389+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:53.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:52.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:53.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:52.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:53.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:52.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:53.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:52.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:53.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:53.116+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:53.378 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:53.116+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:53.378 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:53.116+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:53.378 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:53.118+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:53.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:53.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:53.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:53.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:53.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:53.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:53.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:53.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:54.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:53.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:54.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:53.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:54.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:53.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:54.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:53.796+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:54.352 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:54.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:54.352 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:54.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:54.352 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:54.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:54.352 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:54.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:54.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:54.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:54.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:54.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:54.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:54.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:54.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:54.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:55.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:54.936+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45516 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:49:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:54.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:54.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:54.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:54.774+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:55.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:55.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:55.312 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:55.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:55.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:55.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:55.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:55.099+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:55.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:55.311+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:55.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:55.311+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:55.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:55.311+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:55.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:55.312+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:56.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:55.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:56.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:55.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:56.082 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:55.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:56.082 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:55.797+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:56.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:56.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:56.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:56.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:56.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:56.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:56.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:56.081+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:56.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:56.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:56.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:56.306+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:57.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:56.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:57.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:56.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:57.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:56.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:57.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:56.758+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:57.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:57.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:57.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:57.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:57.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:57.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:57.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:57.287+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:57.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:57.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:57.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:57.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:57.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:57.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:57.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:57.099+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:57.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:57.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:57.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:57.775+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:58.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:58.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:58.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:58.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:58.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:58.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:58.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:58.262+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:58.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:58.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:58.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:58.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:58.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:58.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:58.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:58.125+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:59.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:58.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:59.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:58.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:59.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:58.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:49:59.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:58.810+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:49:59.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:59.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:59.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:59.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:49:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:49:59.253+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:49:59.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:59.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:59.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:59.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:59.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:59.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:49:59.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:49:59.133+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:00.125 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:49:59.942+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45560 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:00.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:59.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:00.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:59.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:00.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:59.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:00.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:49:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:49:59.834+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:00.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:00.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:00.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:00.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:00.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:00.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:00.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:00.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:00.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:00.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:00.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:00.291+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:00.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:00.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:00.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:00.790+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:01.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:01.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:01.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:01.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:01.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:01.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:01.168+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:01.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:01.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:01.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:01.273+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:02.180 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:01.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:02.181 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:01.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:02.181 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:01.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:02.181 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:01.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:02.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:02.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:02.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:02.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:02.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:02.180+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:02.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:02.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:02.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:02.252+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:03.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:02.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:03.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:02.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:03.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:02.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:03.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:02.820+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:03.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:03.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:03.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:03.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:03.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:03.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:03.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:03.284+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:03.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:03.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:03.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:03.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:03.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:03.229+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:03.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:03.230+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:04.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:03.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:04.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:03.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:04.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:03.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:04.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:03.844+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:04.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:04.256+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:04.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:04.256+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:04.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:04.256+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:04.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:04.256+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:04.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:04.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:04.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:04.281+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:05.269 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:04.948+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45606 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:05.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:04.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:05.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:04.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:05.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:04.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:05.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:04.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:05.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:05.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:05.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:05.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:05.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:05.269+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:05.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:05.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:05.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:05.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:05.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:05.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:06.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:05.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:06.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:05.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:06.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:05.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:06.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:05.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:06.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:06.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:06.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:06.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:06.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:06.286+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:06.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:06.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:06.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:06.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:06.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:06.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:06.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:06.329+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:07.246 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:06.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:07.246 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:06.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:07.246 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:06.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:07.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:06.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:07.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:07.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:07.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:07.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:07.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:07.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:07.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:07.294+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:07.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:07.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:07.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:07.246+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:08.261 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:07.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:08.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:07.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:08.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:07.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:08.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:07.821+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:08.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:08.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:08.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:08.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:08.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:08.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:08.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:08.262+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:08.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:08.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:08.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:08.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:09.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:08.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:09.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:08.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:09.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:08.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:09.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:08.797+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:09.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:09.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:09.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:09.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:09.285+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:09.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:09.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:09.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:09.227+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:10.077 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:09.953+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45651 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:10.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:09.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:10.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:09.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:10.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:09.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:10.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:09.806+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:10.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:10.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:10.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:10.184+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:10.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:10.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:10.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:10.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:10.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:10.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:10.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:10.279+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:11.175 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:10.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:11.176 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:10.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:11.176 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:10.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:11.176 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:10.827+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:11.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:11.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:11.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:11.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:11.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:11.175+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:11.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:11.175+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:11.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:11.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:11.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:11.233+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:12.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:11.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:12.207 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:11.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:12.207 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:11.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:12.207 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:11.837+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:12.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:12.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:12.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:12.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:12.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:12.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:12.206+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:12.206+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:12.206+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:12.207+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:13.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:12.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:13.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:12.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:13.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:12.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:13.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:12.858+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:13.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:13.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:13.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:13.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:13.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:13.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:13.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:13.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:13.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:13.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:13.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:13.233+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:14.197 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:13.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:14.197 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:13.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:14.197 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:13.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:14.197 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:13.846+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:14.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:14.196+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:14.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:14.196+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:14.196+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:14.197+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:14.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:14.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:14.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:14.229+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:15.083 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:14.959+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45697 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:15.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:14.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:15.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:14.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:15.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:14.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:15.084 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:14.823+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:15.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:15.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:15.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:15.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:15.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:15.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:15.216+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:15.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:15.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:15.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:15.263+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:16.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:15.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:16.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:15.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:16.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:15.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:16.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:15.818+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:16.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:16.216+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:16.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:16.216+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:16.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:16.216+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:16.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:16.216+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:16.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:16.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:16.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:16.266+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:17.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:16.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:17.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:16.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:17.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:16.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:17.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:16.827+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:17.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:17.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:17.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:17.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:17.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:17.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:17.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:17.174+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:17.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:17.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:17.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:17.241+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:18.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:17.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:18.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:17.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:18.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:17.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:18.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:17.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:18.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:18.236+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:18.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:18.236+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:18.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:18.236+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:18.237+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:18.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:18.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:18.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:18.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:18.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:18.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:18.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:18.212+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:19.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:18.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:19.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:18.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:19.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:18.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:19.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:18.904+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:19.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:19.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:19.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:19.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:19.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:19.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:19.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:19.251+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:19.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:19.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:19.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:19.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:19.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:19.258+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:19.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:19.258+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:20.210 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:19.965+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45741 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:20.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:19.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:20.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:19.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:20.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:19.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:20.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:19.954+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:20.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:20.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:20.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:20.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:20.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:20.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:20.210+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:20.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:20.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:20.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:20.246+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:21.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:21.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:21.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:21.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:21.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:21.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:21.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:21.219+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:21.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:21.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:21.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:21.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:21.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:21.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:21.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:21.178+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:22.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:22.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:22.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:22.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:22.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:22.207+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:22.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:22.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:22.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:22.188+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:22.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:21.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:22.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:21.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:22.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:21.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:22.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:21.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:23.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:23.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:23.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:23.022+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:23.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:23.178+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:23.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:23.178+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:23.296 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:23.178+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:23.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:23.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:23.298 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:23.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:23.299 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:23.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:23.299 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:23.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:23.299 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:23.163+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:24.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:24.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:24.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:24.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:24.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:24.178+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:24.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:24.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:24.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:24.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:24.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:24.147+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:24.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:24.147+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:24.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:24.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:24.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:24.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:24.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:24.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:24.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:24.073+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:25.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:24.971+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45787 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:25.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:25.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:25.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:25.079+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:25.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:25.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:25.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:25.185+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:25.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:25.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:25.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:25.175+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:26.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:26.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:26.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:26.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:26.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:26.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:26.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:26.111+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:26.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:26.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:26.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:26.188+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:26.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:26.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:26.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:26.221+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:27.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:27.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:27.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:27.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:27.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:27.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:27.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:27.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:27.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:27.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:27.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:27.163+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:27.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:27.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:27.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:27.107+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:28.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:28.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:28.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:28.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:28.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:28.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:28.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:28.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:28.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:28.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:28.204+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:28.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:28.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:28.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:28.209+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:29.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:29.181+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:29.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:29.181+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:29.181+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:29.181+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:29.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:29.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:29.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:29.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:29.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:29.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:29.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:29.194+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:29.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:29.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:29.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:29.099+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:30.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:29.977+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45832 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:30.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:30.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:30.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:30.190+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:30.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:30.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:30.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:30.153+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:30.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:30.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:30.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:30.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:30.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:30.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:30.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:30.076+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:31.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:31.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:31.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:31.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:31.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:31.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:31.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:31.122+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:31.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:31.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:31.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:31.194+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:31.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:31.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:31.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:31.102+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:32.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:32.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:32.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:32.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:32.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:32.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:32.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:32.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:32.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:32.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:32.068+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:32.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:32.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:32.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:32.177+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:33.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:33.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:33.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:33.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:33.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:33.111+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:33.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:33.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:33.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:33.224+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:33.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:33.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:33.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:33.143+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:34.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:34.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:34.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:34.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:34.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:34.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:34.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:34.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:34.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:34.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:34.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:34.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:34.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:34.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:34.249+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:35.276 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:34.983+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45878 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:35.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:35.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:35.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:35.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:35.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:35.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:35.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:35.128+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:35.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:35.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:35.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:35.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:35.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:35.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:35.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:35.100+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:35.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:35.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:35.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:35.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:35.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:35.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:35.276+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:36.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:36.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:36.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:36.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:36.108+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:36.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:36.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:36.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:36.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:36.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:36.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:36.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:36.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:37.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:37.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:37.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:37.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:37.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:37.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:37.274+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:37.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:37.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:37.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:37.140+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:37.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:37.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:37.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:37.148+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:38.442 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:38.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:38.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:38.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:38.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:38.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:38.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:38.154+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:38.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:38.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:38.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:38.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:38.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:38.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:38.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:38.313+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:38.444 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:38.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:38.444 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:38.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:38.444 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:38.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:38.444 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:38.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:39.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:39.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:39.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:39.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:39.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:39.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:39.344+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:39.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:39.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:39.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:39.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:39.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:39.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:39.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:39.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:39.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:39.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:39.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:39.159+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:40.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:39.989+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45922 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:40.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:40.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:40.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:40.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:40.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:40.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:40.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:40.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:40.099+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:40.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:40.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:40.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:40.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:40.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:40.313+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:41.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:41.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:41.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:41.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:41.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:41.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:41.234+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:41.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:41.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:41.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:41.278+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:41.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:41.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:41.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:41.069+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:42.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:42.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:42.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:42.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:42.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:42.274+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:42.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:42.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:42.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:42.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:42.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:42.071+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:42.071+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:42.071+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:42.072+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:43.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:43.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:43.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:43.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:43.034+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:43.035+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:43.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:43.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:43.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:43.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:43.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:43.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:43.296+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:43.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:43.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:43.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:43.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:43.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:43.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:43.320+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:44.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:44.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:44.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:44.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:44.010+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:44.011+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:44.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:44.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:44.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:44.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:44.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:44.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:44.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:44.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:44.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:44.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:45.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:44.995+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 45967 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:44.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:44.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:44.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:44.990+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:45.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:45.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:45.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:45.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:45.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:45.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:45.360+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:45.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:45.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:45.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:45.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:46.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:45.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:46.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:45.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:45.992+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:45.993+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:46.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:46.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:46.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:46.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:46.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:46.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:46.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:46.409+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:46.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:46.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:46.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:46.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:47.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:46.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:47.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:46.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:46.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:46.978+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:47.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:47.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:47.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:47.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:47.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:47.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:47.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:47.456+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:47.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:47.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:47.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:47.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:48.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:47.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:48.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:47.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:47.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:47.948+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:48.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:48.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:48.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:48.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:48.506+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:48.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:48.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:48.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:48.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:49.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:48.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:49.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:48.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:49.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:48.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:49.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:48.988+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:49.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:49.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:49.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:49.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:49.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:49.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:49.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:49.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:49.511+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:50.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:50.000+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46013 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:49.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:49.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:49.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:49.979+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:50.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:50.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:50.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:50.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:50.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:50.497+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:50.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:50.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:50.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:50.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:50.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:50.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:50.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:50.396+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:51.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:50.994+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:51.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:50.994+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:50.994+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:50.995+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:51.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:51.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:51.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:51.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:51.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:51.483+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:51.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:51.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:51.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:51.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:52.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:52.015+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:52.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:52.015+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:52.015+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:52.016+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:52.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:52.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:52.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:52.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:52.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:52.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:52.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:52.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:52.444+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:53.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:53.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:53.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:53.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:53.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:53.023+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:53.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:53.024+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:53.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:53.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:53.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:53.285+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:53.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:53.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:53.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:53.478+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:53.478+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:54.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:54.012+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:54.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:54.012+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:54.012+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:54.013+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:54.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:54.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:54.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:54.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:54.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:54.305+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:54.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:54.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:54.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:54.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:54.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:54.490+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:54.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:54.491+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:55.257 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:50:55.006+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46058 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:50:55.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:54.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:55.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:54.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:55.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:54.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:55.258 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:54.997+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:55.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:55.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:55.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:55.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:55.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:55.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:55.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:55.256+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:55.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:55.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:55.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:55.497+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:55.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:55.498+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:56.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:55.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:56.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:55.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:56.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:55.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:56.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:55.966+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:56.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:56.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:56.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:56.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:56.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:56.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:56.462+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:56.463+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:56.463+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:56.464+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:57.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:56.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:57.266 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:56.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:57.266 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:56.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:57.266 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:56.984+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:57.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:57.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:57.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:57.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:57.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:57.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:57.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:57.503+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:57.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:57.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:57.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:57.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:57.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:58.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:57.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:58.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:57.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:57.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:57.986+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:58.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:58.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:58.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:58.226+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:58.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:58.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:58.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:58.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:58.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:58.512+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:50:59.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:59.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:59.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:59.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:59.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:59.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:50:59.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:50:59.008+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:59.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:59.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:59.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:50:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:50:59.180+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:50:59.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:59.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:59.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:59.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:50:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:50:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:50:59.515+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:00.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:00.012+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46103 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:00.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:00.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:00.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:00.170+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:00.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:00.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:00.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:00.039+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:00.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:00.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:01.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:00.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:00.549+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:01.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:01.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:01.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:01.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:01.073+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:01.137+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:01.137+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:01.137+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:01.137+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:02.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:01.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:01.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:01.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:01.563+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:02.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:02.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:02.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:02.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:02.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:02.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:02.106+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:02.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:02.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:02.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:02.097+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:02.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:02.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:03.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:02.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:02.561+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:03.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:03.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:03.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:03.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:03.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:03.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:03.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:03.075+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:03.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:03.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:03.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:03.070+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:04.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:03.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:04.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:03.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:04.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:03.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:04.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:03.543+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:04.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:04.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:04.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:04.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:04.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:04.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:04.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:04.049+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:04.374 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:04.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:04.374 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:04.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:04.375 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:04.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:04.375 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:04.109+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:04.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:04.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:04.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:04.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:04.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:04.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:04.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:04.576+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:05.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:05.018+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46148 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:05.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:05.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:05.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:05.065+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:05.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:05.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:05.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:05.017+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:06.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:05.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:06.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:05.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:06.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:05.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:06.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:05.598+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:06.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:06.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:06.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:06.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:06.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:06.021+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:06.050+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:06.050+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:06.050+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:06.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:06.051+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:06.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:06.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:06.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:06.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:06.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:06.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:06.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:06.557+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:07.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:06.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:07.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:06.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:06.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:06.996+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:07.045+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:07.045+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:07.045+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:07.045+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:08.002 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:07.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:08.002 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:07.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:08.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:07.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:08.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:07.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:08.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:08.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:08.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:08.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:08.002+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:08.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:08.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:08.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:08.056+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:08.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:08.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:08.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:08.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:08.991 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:08.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:08.991 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:08.563+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:09.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:08.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:09.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:08.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:09.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:08.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:09.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:08.990+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:09.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:09.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:09.105+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:09.105+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:09.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:09.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:09.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:09.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:09.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:09.608+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:10.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:10.024+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46194 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:10.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:10.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:10.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:10.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:10.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:10.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:10.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:10.098+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:09.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:09.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:09.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:09.980+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:10.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:10.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:10.950 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:10.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:10.950 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:10.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:10.950 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:10.609+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:11.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:10.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:10.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:10.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:10.949+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:11.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:11.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:11.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:11.090+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:11.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:11.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:11.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:11.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:11.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:11.595+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:11.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:11.596+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:12.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:12.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:12.056+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:12.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:12.057+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:11.929+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:11.929+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:11.929+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:11.930+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:12.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:12.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:12.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:12.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:12.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:12.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:12.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:12.597+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:13.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:12.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:13.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:12.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:13.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:12.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:13.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:12.907+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:13.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:13.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:13.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:13.058+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:13.871 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:13.588+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:13.872 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:13.588+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:13.872 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:13.588+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:13.872 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:13.588+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:14.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:13.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:14.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:13.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:14.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:13.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:14.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:13.871+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:14.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:14.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:14.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:14.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:14.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:14.031+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:14.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:14.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:14.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:14.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:14.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:14.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:14.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:14.587+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:15.151 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:15.030+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46239 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:15.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:14.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:15.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:14.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:15.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:14.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:15.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:14.860+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:15.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:15.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:15.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:15.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:15.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:15.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:15.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:15.014+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:15.883 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:15.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:15.884 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:15.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:15.884 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:15.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:15.884 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:15.597+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:16.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:15.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:16.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:15.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:15.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:15.966+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:16.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:15.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:16.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:15.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:16.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:15.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:16.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:15.883+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:16.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:16.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:16.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:16.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:16.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:16.587+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:16.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:16.588+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:16.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:16.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:16.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:16.932+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:17.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:17.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:17.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:17.011+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:17.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:17.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:17.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:17.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:17.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:17.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:17.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:17.572+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:18.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:17.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:17.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:17.884+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:17.884+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:17.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:17.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:17.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:17.980+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:18.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:18.620+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:18.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:18.620+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:18.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:18.620+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:18.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:18.621+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:19.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:19.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:19.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:19.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:19.010+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:18.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:18.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:18.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:18.901+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:19.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:19.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:19.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:19.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:19.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:20.159 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:20.036+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46284 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:20.159 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:19.852+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:20.159 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:19.852+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:20.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:19.852+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:20.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:19.853+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:20.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:20.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:20.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:20.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:20.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:20.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:20.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:20.003+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:20.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:20.875+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:20.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:20.875+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:20.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:20.875+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:20.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:20.876+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:20.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:20.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:20.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:20.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:20.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:20.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:20.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:20.650+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:21.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:20.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:21.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:20.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:20.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:20.963+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:21.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:21.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:21.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:21.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:21.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:21.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:21.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:21.844+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:21.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:21.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:21.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:21.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:21.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:21.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:21.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:21.644+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:22.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:21.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:22.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:21.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:22.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:21.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:21.949+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:22.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:22.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:22.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:22.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:22.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:22.835+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:22.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:22.836+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:22.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:22.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:22.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:22.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:22.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:22.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:22.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:22.614+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:23.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:22.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:23.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:22.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:22.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:22.985+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:23.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:23.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:23.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:23.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:23.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:23.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:23.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:23.846+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:23.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:23.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:23.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:23.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:23.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:23.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:23.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:23.637+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:24.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:23.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:24.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:23.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:23.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:23.951+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:24.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:24.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:24.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:24.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:24.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:24.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:24.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:24.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:24.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:24.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:24.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:24.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:24.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:24.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:24.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:24.633+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:25.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:25.042+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46329 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:24.935+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:24.935+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:24.935+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:24.936+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:26.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:25.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:25.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:25.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:25.675+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:26.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:25.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:26.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:25.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:25.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:25.891+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:25.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:25.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:25.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:25.872+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:27.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:26.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:26.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:26.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:26.823+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:27.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:26.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:27.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:26.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:26.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:26.890+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:27.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:26.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:27.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:26.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:27.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:26.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:27.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:26.692+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:28.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:27.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:27.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:27.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:27.915+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:27.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:27.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:27.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:27.859+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:27.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:27.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:27.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:27.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:28.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:28.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:28.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:28.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:28.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:28.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:28.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:28.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:28.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:28.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:28.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:28.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:28.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:28.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:28.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:28.657+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:29.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:28.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:28.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:28.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:28.942+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:29.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:29.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:29.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:29.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:29.659+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:30.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:30.047+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46375 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:30.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:29.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:30.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:29.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:30.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:29.802+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:30.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:29.803+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:30.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:29.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:30.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:29.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:30.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:29.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:30.058 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:29.976+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:31.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:30.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:31.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:30.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:31.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:30.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:31.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:30.757+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:31.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:30.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:31.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:30.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:31.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:30.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:31.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:30.704+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:30.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:30.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:30.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:30.952+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:31.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:31.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:31.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:31.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:31.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:31.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:31.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:31.750+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:31.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:31.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:31.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:31.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:31.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:31.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:31.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:31.683+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:32.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:31.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:31.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:31.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:31.984+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:32.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:32.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:32.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:32.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:32.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:32.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:32.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:32.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:32.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:32.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:32.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:32.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:32.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:32.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:32.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:32.643+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:32.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:32.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:32.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:32.948+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:33.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:33.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:33.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:33.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:33.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:33.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:33.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:33.740+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:33.919 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:33.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:33.919 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:33.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:33.919 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:33.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:33.919 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:33.646+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:34.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:33.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:33.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:33.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:33.917+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:34.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:34.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:34.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:34.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:34.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:34.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:34.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:34.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:34.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:34.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:34.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:34.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:34.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:34.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:34.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:34.608+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:35.175 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:35.054+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46420 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:35.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:34.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:35.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:34.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:35.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:34.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:35.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:34.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:35.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:35.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:35.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:35.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:35.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:35.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:35.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:35.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:35.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:35.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:35.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:35.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:35.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:35.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:35.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:35.626+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:36.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:35.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:36.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:35.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:35.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:35.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:36.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:36.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:36.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:36.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:36.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:36.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:36.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:36.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:36.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:36.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:36.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:36.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:36.651+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:37.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:36.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:36.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:36.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:36.910+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:38.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:37.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:37.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:37.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:37.702+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:37.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:37.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:37.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:37.682+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:37.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:37.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:37.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:37.864+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:38.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:38.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:38.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:38.684+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:38.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:38.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:38.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:38.715+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:38.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:38.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:38.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:39.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:38.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:39.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:39.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:39.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:39.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:39.680+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:39.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:39.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:39.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:39.666+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:40.182 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:40.059+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46465 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:40.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:39.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:40.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:39.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:40.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:39.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:40.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:39.859+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:41.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:40.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:40.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:40.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:40.644+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:40.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:40.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:40.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:40.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:40.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:40.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:41.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:40.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:41.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:40.834+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:41.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:41.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:41.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:41.642+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:41.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:41.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:41.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:41.843+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:42.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:41.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:42.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:41.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:41.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:42.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:41.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:43.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:42.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:42.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:42.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:42.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:42.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:42.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:43.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:42.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:43.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:42.635+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:42.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:42.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:42.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:42.801+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:44.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:43.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:44.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:43.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:44.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:43.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:44.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:43.849+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:43.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:43.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:43.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:43.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:43.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:43.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:43.659+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:43.660+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:45.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:44.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:44.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:44.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:44.702+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:44.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:44.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:44.660+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:45.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:44.661+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:44.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:44.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:44.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:44.816+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:45.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:45.066+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46510 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:46.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:45.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:46.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:45.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:46.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:45.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:46.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:45.684+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:45.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:46.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:45.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:46.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:45.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:46.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:45.658+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:45.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:45.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:46.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:45.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:46.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:45.805+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:46.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:46.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:46.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:46.639+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:46.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:46.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:46.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:46.758+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:46.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:46.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:46.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:46.652+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:48.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:47.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:47.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:47.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:47.619+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:47.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:47.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:47.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:47.743+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:48.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:47.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:48.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:47.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:48.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:47.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:48.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:47.680+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:49.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:48.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:48.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:48.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:48.632+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:48.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:48.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:48.656+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:48.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:48.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:48.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:48.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:48.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:49.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:49.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:49.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:49.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:49.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:49.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:49.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:49.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:49.663+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:49.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:49.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:49.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:49.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:50.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:50.072+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46556 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:51.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:50.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:50.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:50.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:50.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:50.678+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:50.678+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:50.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:50.679+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:50.736+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:51.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:51.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:51.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:51.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:52.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:51.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:51.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:51.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:51.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:51.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:51.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:52.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:51.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:52.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:51.681+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:52.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:52.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:52.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:52.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:53.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:52.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:53.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:52.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:53.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:52.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:53.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:52.684+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:53.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:52.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:53.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:52.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:53.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:52.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:53.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:52.699+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:54.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:53.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:54.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:53.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:53.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:53.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:54.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:53.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:54.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:53.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:54.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:53.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:54.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:53.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:53.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:53.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:54.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:53.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:54.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:53.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:55.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:54.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:55.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:54.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:55.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:54.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:55.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:54.701+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:54.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:54.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:54.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:54.672+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:55.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:54.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:55.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:54.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:55.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:54.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:55.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:54.630+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:55.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:51:55.078+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46601 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:51:56.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:55.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:56.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:55.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:56.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:55.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:56.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:55.673+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:56.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:55.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:56.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:55.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:56.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:55.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:56.045 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:55.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:56.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:55.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:56.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:55.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:56.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:55.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:56.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:55.682+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:56.729+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:56.729+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:56.729+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:56.730+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:57.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:56.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:57.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:56.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:57.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:56.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:57.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:56.702+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:56.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:57.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:56.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:57.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:56.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:57.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:56.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:57.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:57.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:57.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:57.778+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:58.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:57.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:58.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:57.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:58.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:57.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:58.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:57.636+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:58.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:57.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:58.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:57.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:58.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:57.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:58.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:57.721+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:59.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:58.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:58.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:58.744+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:58.744+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:51:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:58.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:58.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:58.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:59.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:58.684+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:58.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:58.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:58.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:58.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:59.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:59.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:59.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:59.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:51:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:51:59.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:59.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:59.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:59.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:51:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:51:59.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:51:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:59.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:59.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:59.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:51:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:51:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:51:59.699+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:00.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:00.084+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46646 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:00.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:00.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:00.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:00.654+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:00.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:00.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:00.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:00.716+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:00.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:00.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:00.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:01.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:00.640+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:02.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:01.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:01.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:01.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:01.622+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:02.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:01.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:02.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:01.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:01.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:01.685+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:01.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:01.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:01.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:01.660+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:02.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:02.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:02.613+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:02.613+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:02.673+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:02.673+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:02.673+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:02.674+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:03.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:02.610+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:03.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:02.610+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:03.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:02.610+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:03.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:02.611+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:04.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:03.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:03.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:03.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:03.645+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:03.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:04.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:03.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:04.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:03.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:04.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:03.622+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:03.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:03.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:03.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:04.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:03.601+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:05.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:04.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:04.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:04.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:04.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:05.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:04.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:05.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:04.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:05.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:04.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:05.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:04.602+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:04.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:04.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:04.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:04.616+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:05.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:05.090+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46691 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:05.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:05.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:05.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:05.583+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:06.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:05.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:06.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:05.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:06.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:05.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:06.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:05.570+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:05.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:05.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:06.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:05.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:06.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:05.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:06.619+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:06.619+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:06.619+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:06.619+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:07.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:06.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:07.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:06.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:07.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:06.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:07.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:06.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:06.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:06.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:07.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:06.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:07.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:06.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:07.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:07.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:07.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:07.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:07.677+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:07.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:07.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:07.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:07.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:07.641+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:07.642+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:07.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:07.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:07.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:07.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:08.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:08.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:08.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:08.677+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:08.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:08.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:08.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:08.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:08.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:08.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:08.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:08.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:08.652+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:09.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:09.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:09.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:09.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:09.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:09.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:09.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:09.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:09.647+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:09.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:09.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:09.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:09.494+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:10.480 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:10.097+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46737 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:10.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:10.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:10.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:10.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:10.641+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:10.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:10.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:10.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:10.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:10.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:10.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:10.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:10.605+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:10.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:10.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:10.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:10.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:10.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:10.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:10.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:10.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:11.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:11.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:11.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:11.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:11.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:11.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:11.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:11.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:11.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:11.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:11.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:11.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:11.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:11.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:11.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:11.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:11.643+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:12.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:12.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:12.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:12.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:12.663+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:12.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:12.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:12.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:12.580+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:12.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:12.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:12.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:12.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:12.485+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:13.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:13.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:13.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:13.624+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:13.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:13.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:13.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:13.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:13.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:13.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:13.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:13.713+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:13.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:13.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:13.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:13.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:13.525+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:14.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:14.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:14.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:14.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:14.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:14.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:14.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:14.708+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:14.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:14.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:14.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:14.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:14.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:14.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:14.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:14.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:15.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:15.103+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46782 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:15.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:15.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:15.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:15.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:15.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:15.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:15.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:15.673+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:15.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:15.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:15.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:16.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:15.558+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:16.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:16.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:16.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:16.704+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:16.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:16.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:16.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:16.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:16.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:16.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:16.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:16.579+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:17.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:17.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:17.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:17.717+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:17.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:17.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:17.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:17.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:17.621+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:17.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:17.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:17.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:17.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:18.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:18.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:18.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:18.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:18.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:18.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:18.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:18.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:18.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:18.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:18.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:18.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:18.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:18.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:18.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:18.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:19.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:18.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:19.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:18.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:19.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:18.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:19.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:18.758+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:19.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:19.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:19.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:19.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:19.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:19.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:19.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:19.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:19.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:19.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:19.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:19.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:19.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:19.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:19.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:19.690+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:20.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:19.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:20.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:19.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:20.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:19.783+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:20.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:19.784+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:20.467 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:20.109+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46827 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:20.729 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:20.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:20.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:20.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:20.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:20.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:20.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:20.466+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:21.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:20.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:21.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:20.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:21.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:20.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:21.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:20.729+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:21.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:20.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:21.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:20.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:21.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:20.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:21.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:20.779+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:21.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:21.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:21.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:21.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:21.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:21.501+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:21.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:21.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:21.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:21.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:21.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:21.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:21.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:21.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:21.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:21.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:22.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:21.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:22.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:21.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:22.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:21.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:22.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:21.779+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:22.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:22.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:22.789 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:22.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:22.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:22.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:22.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:22.492+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:22.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:22.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:22.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:22.789+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:23.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:22.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:22.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:22.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:22.801+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:23.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:23.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:23.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:23.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:23.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:23.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:23.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:23.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:23.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:23.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:23.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:23.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:24.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:23.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:23.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:23.784+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:23.785+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:24.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:24.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:24.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:24.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:24.503+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:25.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:24.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:25.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:24.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:25.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:24.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:25.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:24.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:25.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:24.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:25.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:24.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:25.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:24.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:25.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:24.822+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:25.536 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:25.116+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46872 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:25.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:25.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:25.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:25.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:25.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:25.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:25.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:25.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:25.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:25.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:25.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:25.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:25.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:25.536+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:26.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:25.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:25.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:25.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:25.833+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:26.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:26.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:26.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:26.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:26.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:26.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:26.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:26.758+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:26.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:26.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:26.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:26.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:26.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:26.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:26.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:26.575+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:27.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:26.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:26.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:26.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:26.867+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:28.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:27.620+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:27.620+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:27.620+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:27.620+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:27.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:27.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:27.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:27.773+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:27.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:27.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:27.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:27.822+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:29.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:28.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:29.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:28.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:29.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:28.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:29.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:28.625+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:28.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:28.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:28.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:28.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:28.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:28.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:28.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:28.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:29.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:29.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:29.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:29.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:29.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:29.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:29.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:29.583+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:29.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:29.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:29.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:29.751+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:30.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:29.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:30.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:29.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:30.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:29.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:30.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:29.850+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:30.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:30.122+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46918 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:30.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:30.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:30.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:30.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:30.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:30.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:30.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:30.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:30.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:30.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:30.882 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:30.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:30.882 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:30.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:30.882 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:30.714+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:31.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:30.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:30.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:30.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:30.880+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:31.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:31.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:31.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:31.710+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:31.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:31.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:31.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:31.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:32.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:31.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:31.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:31.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:31.902+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:32.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:32.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:32.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:32.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:32.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:32.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:32.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:32.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:32.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:32.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:32.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:32.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:32.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:32.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:32.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:32.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:33.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:32.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:32.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:32.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:32.876+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:33.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:33.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:33.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:33.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:33.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:33.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:33.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:33.592+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:33.845 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:33.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:33.845 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:33.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:33.846 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:33.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:33.846 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:33.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:33.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:33.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:33.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:33.844+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:35.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:34.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:35.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:34.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:35.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:34.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:35.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:34.724+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:35.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:34.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:35.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:34.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:35.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:34.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:35.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:34.819+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:34.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:34.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:34.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:34.603+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:35.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:35.128+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 46963 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:36.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:35.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:35.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:35.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:35.575+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:35.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:36.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:35.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:35.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:35.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:36.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:35.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:36.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:35.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:36.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:35.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:36.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:35.800+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:36.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:36.783+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:36.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:36.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:36.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:36.755+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:36.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:36.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:36.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:36.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:38.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:37.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:37.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:37.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:37.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:37.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:37.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:37.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:37.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:37.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:37.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:37.612+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:37.612+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:38.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:38.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:38.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:38.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:38.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:38.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:38.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:38.710+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:38.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:38.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:38.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:38.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:38.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:38.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:38.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:38.638+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:38.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:38.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:38.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:38.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:38.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:38.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:38.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:38.823+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:39.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:39.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:39.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:39.624+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:39.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:39.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:39.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:39.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:40.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:39.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:40.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:39.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:40.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:39.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:40.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:39.846+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:40.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:40.134+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47008 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:40.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:40.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:40.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:40.639+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:40.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:40.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:40.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:40.711+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:40.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:40.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:41.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:40.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:41.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:40.844+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:42.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:41.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:41.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:41.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:41.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:41.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:41.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:42.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:41.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:42.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:41.798+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:41.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:41.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:41.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:41.740+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:42.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:42.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:42.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:42.813+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:42.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:42.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:42.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:42.680+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:42.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:42.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:43.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:42.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:43.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:42.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:44.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:43.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:43.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:43.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:43.664+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:44.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:43.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:43.632+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:43.632+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:43.632+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:43.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:43.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:43.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:43.854+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:44.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:44.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:44.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:44.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:44.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:44.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:44.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:44.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:44.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:44.633+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:44.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:44.633+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:44.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:44.633+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:44.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:44.634+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:45.162 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:45.140+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47053 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:45.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:44.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:45.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:44.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:45.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:44.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:45.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:44.901+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:45.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:45.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:45.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:45.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:45.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:45.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:45.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:45.616+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:45.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:45.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:45.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:45.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:45.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:45.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:45.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:45.625+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:45.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:45.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:45.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:45.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:45.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:45.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:46.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:45.920+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:46.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:46.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:46.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:46.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:46.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:46.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:46.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:46.626+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:46.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:46.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:46.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:46.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:46.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:46.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:46.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:46.578+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:47.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:46.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:46.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:46.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:46.961+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:47.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:47.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:47.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:47.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:47.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:47.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:47.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:47.663+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:47.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:47.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:47.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:47.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:47.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:47.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:47.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:47.543+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:48.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:47.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:47.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:47.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:47.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:48.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:48.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:48.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:48.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:48.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:48.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:48.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:48.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:48.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:48.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:48.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:48.942+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:49.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:49.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:49.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:49.653+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:49.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:49.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:49.476+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:49.477+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:50.269 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:50.146+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47100 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:50.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:49.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:50.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:49.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:50.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:49.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:50.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:49.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:50.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:50.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:50.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:50.508+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:51.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:50.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:50.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:50.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:50.903+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:51.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:50.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:50.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:50.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:50.677+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:51.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:51.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:51.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:51.717+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:51.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:51.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:51.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:51.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:51.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:51.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:51.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:51.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:52.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:52.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:52.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:52.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:52.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:52.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:52.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:52.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:52.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:52.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:52.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:52.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:52.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:52.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:52.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:52.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:52.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:52.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:52.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:52.896+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:53.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:53.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:53.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:53.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:53.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:53.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:53.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:53.737+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:53.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:53.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:53.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:53.532+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:54.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:53.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:53.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:53.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:53.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:54.853 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:54.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:54.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:54.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:54.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:54.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:54.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:54.759+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:54.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:54.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:54.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:54.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:54.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:54.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:54.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:54.572+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:54.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:54.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:54.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:54.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:54.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:54.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:55.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:54.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:55.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:52:55.152+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47145 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:52:55.875 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:55.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:55.876 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:55.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:55.876 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:55.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:55.876 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:55.543+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:55.876 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:55.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:55.876 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:55.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:55.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:55.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:55.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:55.732+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:56.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:55.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:55.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:55.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:55.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:56.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:56.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:56.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:56.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:56.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:56.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:56.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:56.574+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:56.910 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:56.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:56.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:56.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:56.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:56.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:56.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:56.710+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:56.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:56.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:56.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:56.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:57.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:57.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:57.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:57.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:57.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:57.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:57.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:57.685+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:57.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:57.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:57.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:57.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:57.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:57.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:57.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:57.592+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:57.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:57.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:57.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:57.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:58.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:58.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:58.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:58.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:58.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:58.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:58.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:58.666+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:58.849 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:58.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:58.849 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:58.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:58.849 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:58.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:58.849 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:58.578+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:58.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:58.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:58.847+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:52:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:58.847+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:52:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:59.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:59.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:59.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:52:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:52:59.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:52:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:59.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:59.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:59.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:52:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:52:59.626+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:00.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:59.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:00.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:59.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:00.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:59.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:00.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:52:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:52:59.836+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:00.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:00.158+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47190 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:00.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:00.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:00.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:00.866+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:00.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:00.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:00.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:00.645+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:00.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:00.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:00.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:01.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:00.664+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:02.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:01.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:02.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:01.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:02.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:01.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:02.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:01.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:01.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:01.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:01.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:02.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:01.689+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:02.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:01.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:02.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:01.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:02.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:01.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:02.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:01.908+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:02.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:02.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:02.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:02.873+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:02.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:02.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:02.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:02.658+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:02.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:02.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:02.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:02.725+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:03.621+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:03.621+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:03.621+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:04.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:03.622+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:04.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:03.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:03.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:03.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:03.841+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:03.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:03.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:03.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:03.705+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:04.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:04.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:04.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:04.680+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:04.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:04.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:04.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:04.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:05.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:04.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:05.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:04.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:05.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:04.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:05.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:04.626+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:05.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:05.165+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47236 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:05.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:05.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:05.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:05.651+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:05.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:05.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:05.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:05.710+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:05.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:05.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:06.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:05.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:06.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:05.828+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:06.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:06.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:06.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:07.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:06.690+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:06.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:06.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:06.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:06.693+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:07.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:06.793+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:07.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:06.793+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:07.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:06.793+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:07.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:06.794+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:07.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:07.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:07.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:07.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:07.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:07.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:07.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:07.717+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:07.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:07.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:08.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:07.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:08.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:07.819+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:08.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:08.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:08.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:08.702+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:09.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:08.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:09.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:08.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:09.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:08.726+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:09.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:08.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:09.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:08.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:09.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:08.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:08.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:09.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:08.844+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:09.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:09.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:09.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:09.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:09.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:09.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:09.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:09.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:09.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:09.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:09.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:10.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:09.775+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:09.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:09.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:09.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:09.892+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:10.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:10.171+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47281 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:10.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:10.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:10.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:10.869+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:11.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:10.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:11.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:10.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:11.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:10.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:11.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:10.737+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:10.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:10.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:11.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:10.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:11.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:10.688+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:11.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:11.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:11.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:11.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:11.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:11.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:11.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:11.835+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:11.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:11.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:12.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:11.719+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:12.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:11.719+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:12.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:12.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:12.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:12.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:12.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:12.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:12.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:12.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:12.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:12.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:12.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:13.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:12.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:13.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:13.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:13.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:13.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:14.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:13.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:14.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:13.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:14.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:13.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:14.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:13.829+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:13.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:13.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:13.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:14.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:13.791+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:15.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:14.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:15.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:14.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:15.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:14.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:15.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:14.808+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:14.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:14.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:14.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:14.607+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:14.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:14.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:14.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:15.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:14.781+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:15.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:15.177+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47326 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:16.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:15.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:16.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:15.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:16.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:15.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:16.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:15.787+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:16.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:15.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:15.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:15.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:16.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:15.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:15.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:15.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:15.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:15.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:16.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:16.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:16.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:16.644+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:16.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:16.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:16.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:16.739+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:16.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:16.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:16.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:16.808+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:18.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:17.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:18.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:17.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:18.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:17.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:18.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:17.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:17.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:17.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:17.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:17.670+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:17.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:17.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:17.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:17.836+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:19.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:18.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:19.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:18.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:19.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:18.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:19.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:18.761+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:18.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:18.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:18.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:18.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:18.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:18.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:18.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:19.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:18.805+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:19.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:19.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:19.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:19.728+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:19.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:19.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:19.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:19.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:20.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:19.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:20.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:19.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:20.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:19.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:20.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:19.793+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:20.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:20.183+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47371 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:20.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:20.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:20.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:21.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:20.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:21.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:20.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:20.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:20.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:20.648+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:21.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:20.713+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:21.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:20.713+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:21.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:20.713+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:21.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:20.714+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:21.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:21.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:21.808+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:21.809+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:21.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:21.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:21.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:21.694+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:21.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:21.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:21.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:22.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:21.707+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:22.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:22.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:22.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:23.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:22.801+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:22.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:22.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:22.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:22.743+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:22.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:22.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:22.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:23.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:22.738+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:24.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:23.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:24.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:23.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:24.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:23.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:24.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:23.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:23.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:23.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:23.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:23.767+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:23.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:23.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:23.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:24.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:23.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:24.858+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:24.858+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:24.858+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:24.859+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:24.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:24.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:24.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:24.736+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:25.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:24.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:25.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:24.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:25.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:24.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:25.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:24.722+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:25.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:25.190+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47418 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:25.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:25.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:25.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:26.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:25.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:25.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:26.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:25.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:26.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:25.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:26.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:25.691+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:25.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:25.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:25.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:26.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:25.882+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:27.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:26.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:27.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:26.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:26.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:26.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:27.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:26.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:27.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:26.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:27.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:26.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:27.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:26.757+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:26.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:26.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:26.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:27.045 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:26.735+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:27.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:27.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:27.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:27.866+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:27.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:27.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:27.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:27.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:27.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:27.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:28.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:27.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:28.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:27.727+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:28.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:28.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:28.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:28.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:28.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:28.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:28.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:29.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:28.720+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:28.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:28.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:28.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:29.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:28.878+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:29.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:29.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:29.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:29.701+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:29.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:29.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:29.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:29.655+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:30.197 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:29.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:30.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:29.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:30.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:29.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:30.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:29.914+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:30.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:30.196+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47463 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:30.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:30.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:30.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:30.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:30.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:30.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:30.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:30.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:30.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:30.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:30.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:30.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:30.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:30.673+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:30.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:30.674+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:30.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:30.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:30.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:30.944+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:31.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:31.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:31.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:31.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:31.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:31.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:31.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:31.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:31.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:31.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:31.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:31.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:31.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:31.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:31.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:31.700+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:31.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:31.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:31.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:31.963+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:32.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:32.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:32.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:32.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:32.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:32.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:32.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:32.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:32.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:32.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:32.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:32.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:32.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:32.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:32.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:32.655+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:32.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:32.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:32.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:32.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:33.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:33.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:33.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:33.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:33.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:33.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:33.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:33.565+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:33.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:33.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:33.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:33.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:33.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:33.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:33.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:33.687+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:33.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:33.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:33.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:33.960+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:35.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:34.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:35.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:34.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:35.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:34.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:35.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:34.562+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:35.000 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:34.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:35.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:34.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:35.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:34.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:35.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:34.646+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:35.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:35.202+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47509 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:34.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:34.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:34.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:34.999+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:35.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:35.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:35.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:35.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:35.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:35.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:35.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:35.569+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:35.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:35.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:35.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:35.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:35.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:35.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:35.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:35.651+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:36.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:35.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:35.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:35.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:35.961+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:36.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:36.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:36.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:36.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:36.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:36.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:36.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:36.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:36.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:36.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:36.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:36.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:36.918 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:36.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:36.918 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:36.608+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:36.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:36.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:36.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:36.915+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:37.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:37.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:37.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:37.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:37.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:37.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:37.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:37.554+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:37.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:37.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:37.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:37.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:37.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:37.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:37.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:37.621+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:38.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:37.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:38.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:37.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:38.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:37.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:38.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:37.957+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:38.971 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:38.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:38.971 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:38.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:38.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:38.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:38.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:38.551+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:38.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:38.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:38.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:38.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:38.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:38.650+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:38.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:38.651+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:38.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:38.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:38.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:38.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:39.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:39.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:39.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:39.610+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:39.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:39.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:39.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:39.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:40.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:40.208+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47554 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:39.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:39.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:39.989+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:40.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:39.989+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:40.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:40.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:40.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:40.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:40.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:40.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:40.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:40.602+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:40.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:40.610+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:40.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:40.610+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:40.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:40.610+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:40.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:40.610+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:40.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:40.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:40.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:40.943+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:41.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:41.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:41.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:41.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:41.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:41.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:41.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:41.555+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:41.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:41.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:41.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:41.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:41.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:41.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:41.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:41.593+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:41.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:41.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:41.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:41.960+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:43.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:42.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:43.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:42.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:43.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:42.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:43.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:42.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:43.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:42.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:43.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:42.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:43.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:42.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:43.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:42.585+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:43.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:43.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:43.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:43.002+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:43.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:43.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:43.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:43.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:43.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:43.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:43.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:43.621+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:43.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:43.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:43.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:43.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:43.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:43.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:43.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:43.548+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:43.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:43.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:43.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:43.960+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:44.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:44.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:44.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:44.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:44.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:44.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:44.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:44.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:44.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:44.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:44.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:44.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:44.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:44.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:44.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:44.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:45.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:44.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:45.216 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:44.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:45.216 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:44.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:45.216 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:44.929+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:45.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:45.214+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47602 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:45.899 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:45.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:45.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:45.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:45.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:45.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:45.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:45.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:45.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:45.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:45.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:45.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:45.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:45.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:45.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:45.544+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:45.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:45.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:45.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:45.898+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:46.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:46.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:46.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:46.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:46.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:46.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:46.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:46.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:46.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:46.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:46.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:46.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:46.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:46.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:46.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:46.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:46.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:46.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:46.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:46.863+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:47.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:47.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:47.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:47.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:47.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:47.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:47.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:47.588+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:47.859 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:47.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:47.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:47.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:47.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:47.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:47.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:47.548+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:47.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:47.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:47.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:47.858+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:48.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:48.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:48.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:48.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:48.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:48.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:48.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:48.538+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:48.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:48.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:48.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:49.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:48.861+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:49.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:49.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:49.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:49.530+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:49.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:49.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:49.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:49.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:50.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:49.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:50.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:49.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:50.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:49.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:50.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:49.874+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:50.504 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:50.220+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47647 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:50.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:50.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:50.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:50.503+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:50.795 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:50.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:50.795 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:50.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:50.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:50.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:50.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:50.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:50.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:50.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:50.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:50.857+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:51.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:51.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:51.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:51.581+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:51.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:51.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:51.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:51.499+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:51.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:51.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:51.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:51.881+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:52.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:52.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:52.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:52.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:52.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:52.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:52.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:52.601+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:52.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:52.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:52.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:52.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:53.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:53.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:53.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:53.492+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:53.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:53.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:53.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:53.563+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:53.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:53.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:53.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:53.827+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:54.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:54.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:54.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:54.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:54.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:54.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:54.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:54.516+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:55.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:54.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:55.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:54.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:55.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:54.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:55.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:54.877+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:55.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:53:55.226+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47692 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:53:55.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:55.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:55.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:55.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:55.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:55.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:55.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:55.599+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:55.864 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:55.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:55.864 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:55.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:55.864 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:55.563+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:55.864 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:55.563+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:55.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:55.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:55.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:55.862+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:56.909 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:56.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:56.909 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:56.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:56.910 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:56.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:56.910 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:56.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:56.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:56.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:56.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:56.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:56.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:56.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:56.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:56.551+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:56.908+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:56.908+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:56.908+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:56.908+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:57.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:57.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:57.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:57.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:57.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:57.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:57.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:57.584+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:57.906 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:57.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:57.906 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:57.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:57.906 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:57.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:57.906 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:57.545+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:57.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:57.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:57.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:57.904+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:58.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:58.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:58.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:58.504+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:58.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:58.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:58.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:58.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:58.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:58.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:58.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:53:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:58.905+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:53:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:59.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:59.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:59.476+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:53:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:53:59.477+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:53:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:59.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:59.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:59.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:53:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:53:59.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:00.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:59.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:00.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:59.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:00.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:59.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:00.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:53:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:53:59.921+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:00.488 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:00.232+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47737 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:00.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:00.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:00.487+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:00.488+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:00.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:00.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:00.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:00.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:00.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:00.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:00.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:00.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:01.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:01.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:01.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:01.537+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:01.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:01.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:01.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:01.566+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:01.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:01.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:01.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:01.945+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:02.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:02.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:02.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:02.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:02.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:02.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:02.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:02.505+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:02.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:02.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:02.600+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:02.600+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:02.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:02.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:02.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:02.943+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:03.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:03.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:03.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:03.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:03.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:03.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:03.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:03.494+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:03.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:03.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:03.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:03.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:03.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:03.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:03.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:03.915+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:04.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:04.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:04.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:04.623+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:04.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:04.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:04.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:04.512+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:05.240 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:04.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:05.241 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:04.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:05.241 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:04.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:05.241 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:04.933+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:05.520 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:05.239+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47783 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:05.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:05.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:05.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:05.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:05.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:05.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:05.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:05.519+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:05.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:05.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:05.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:05.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:06.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:06.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:06.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:06.506+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:06.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:06.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:06.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:06.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:06.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:06.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:06.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:07.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:06.934+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:07.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:07.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:07.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:07.584+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:07.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:07.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:07.537+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:07.538+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:07.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:07.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:07.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:07.959+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:08.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:08.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:08.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:08.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:08.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:08.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:08.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:08.494+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:08.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:08.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:08.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:08.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:08.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:08.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:08.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:08.623+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:08.981+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:08.981+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:08.981+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:08.982+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:09.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:09.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:09.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:09.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:09.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:09.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:09.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:09.530+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:10.255 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:10.245+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47828 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:10.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:09.995+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:10.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:09.995+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:10.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:09.995+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:10.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:09.996+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:10.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:10.531+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:10.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:10.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:10.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:10.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:10.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:10.533+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:10.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:10.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:10.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:10.662+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:10.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:10.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:10.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:10.992+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:11.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:11.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:11.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:11.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:11.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:11.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:11.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:11.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:11.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:11.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:11.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:11.528+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:11.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:11.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:11.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:11.965+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:13.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:12.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:13.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:12.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:13.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:12.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:13.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:12.662+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:13.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:12.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:13.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:12.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:13.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:12.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:13.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:12.576+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:13.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:13.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:13.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:13.006+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:14.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:13.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:14.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:13.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:14.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:13.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:14.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:13.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:14.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:13.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:14.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:13.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:14.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:13.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:14.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:13.612+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:14.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:14.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:14.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:14.022+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:14.989 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:14.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:14.990 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:14.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:14.990 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:14.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:14.990 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:14.655+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:14.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:14.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:14.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:14.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:14.991 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:14.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:14.991 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:14.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:15.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:14.988+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:15.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:14.988+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:15.253 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:14.988+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:15.253 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:14.988+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:15.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:15.251+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47873 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:15.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:15.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:15.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:15.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:15.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:15.651+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:15.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:15.652+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:15.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:15.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:15.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:15.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:15.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:15.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:15.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:15.646+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:15.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:15.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:15.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:15.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:16.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:16.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:16.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:16.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:16.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:16.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:16.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:16.652+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:16.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:16.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:16.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:16.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:16.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:16.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:16.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:16.695+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:16.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:16.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:16.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:16.932+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:17.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:17.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:17.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:17.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:17.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:17.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:17.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:17.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:17.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:17.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:17.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:17.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:17.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:17.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:17.950 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:17.693+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:17.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:17.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:17.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:17.948+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:18.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:18.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:18.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:18.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:18.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:18.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:18.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:18.653+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:18.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:18.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:18.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:18.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:18.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:18.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:18.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:18.654+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:18.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:18.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:18.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:18.930+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:19.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:19.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:19.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:19.657+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:19.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:19.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:19.661+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:19.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:20.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:19.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:20.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:19.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:20.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:19.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:20.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:19.957+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:20.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:20.257+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47917 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:20.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:20.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:20.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:20.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:20.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:20.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:20.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:20.632+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:20.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:20.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:20.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:20.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:20.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:20.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:20.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:20.703+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:20.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:20.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:20.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:20.958+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:21.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:21.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:21.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:21.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:21.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:21.592+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:21.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:21.593+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:21.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:21.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:21.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:21.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:21.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:21.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:21.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:21.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:21.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:21.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:21.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:21.946+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:22.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:22.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:22.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:22.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:22.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:22.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:22.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:22.715+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:22.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:22.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:22.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:22.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:22.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:22.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:22.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:22.579+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:22.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:22.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:22.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:22.957+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:23.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:23.624+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:23.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:23.624+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:23.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:23.624+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:23.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:23.625+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:23.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:23.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:23.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:23.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:23.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:23.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:23.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:23.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:23.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:23.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:23.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:23.937+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:24.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:24.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:24.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:24.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:24.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:24.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:24.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:24.745+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:24.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:24.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:24.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:24.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:24.905 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:24.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:24.905 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:24.580+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:25.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:24.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:25.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:24.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:25.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:24.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:25.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:24.903+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:25.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:25.264+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 47960 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:25.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:25.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:25.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:25.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:25.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:25.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:25.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:25.717+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:25.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:25.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:25.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:25.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:25.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:25.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:25.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:25.543+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:25.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:25.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:25.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:25.893+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:26.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:26.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:26.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:26.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:26.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:26.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:26.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:26.704+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:26.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:26.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:26.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:26.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:26.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:26.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:26.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:26.585+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:26.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:26.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:26.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:26.939+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:27.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:27.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:27.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:27.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:27.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:27.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:27.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:27.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:27.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:27.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:27.905 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:27.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:27.905 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:27.578+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:27.905 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:27.579+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:28.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:27.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:28.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:27.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:28.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:27.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:28.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:27.903+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:28.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:28.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:28.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:28.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:28.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:28.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:28.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:28.679+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:28.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:28.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:28.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:28.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:28.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:28.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:28.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:28.606+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:28.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:28.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:28.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:28.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:29.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:29.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:29.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:29.667+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:29.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:29.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:29.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:29.609+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:30.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:29.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:30.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:29.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:30.272 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:29.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:30.272 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:29.876+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:30.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:30.270+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48004 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:30.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:30.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:30.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:30.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:30.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:30.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:30.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:30.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:30.862 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:30.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:30.862 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:30.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:30.862 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:30.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:30.862 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:30.577+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:30.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:30.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:30.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:30.860+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:31.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:31.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:31.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:31.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:31.909 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:31.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:31.909 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:31.755+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:31.909 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:31.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:31.909 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:31.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:31.909 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:31.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:31.909 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:31.549+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:31.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:31.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:31.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:31.907+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:32.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:32.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:32.520+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:32.520+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:32.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:32.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:32.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:32.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:32.767+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:32.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:32.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:32.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:32.882+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:33.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:33.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:33.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:33.483+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:33.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:33.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:33.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:33.853+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:33.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:33.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:33.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:33.809+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:34.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:34.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:34.492+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:34.493+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:35.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:34.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:35.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:34.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:35.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:34.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:35.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:34.817+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:35.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:34.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:35.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:34.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:35.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:34.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:35.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:34.876+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:35.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:35.276+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48050 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:35.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:35.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:35.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:35.470+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:35.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:35.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:35.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:35.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:35.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:35.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:35.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:35.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:36.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:36.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:36.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:36.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:36.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:36.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:36.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:36.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:36.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:36.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:36.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:36.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:36.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:36.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:36.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:36.929+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:37.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:37.413+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:37.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:37.413+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:37.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:37.413+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:37.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:37.414+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:37.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:37.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:37.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:38.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:37.917+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:37.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:37.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:37.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:37.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:38.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:38.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:38.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:38.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:38.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:38.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:38.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:38.371+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:38.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:38.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:38.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:38.907+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:39.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:38.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:38.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:38.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:38.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:39.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:39.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:39.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:39.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:39.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:39.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:39.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:39.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:39.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:39.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:39.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:40.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:39.887+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:39.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:39.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:39.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:39.759+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:40.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:40.282+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48094 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:40.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:40.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:40.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:40.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:40.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:40.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:40.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:40.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:40.844+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:40.844+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:40.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:40.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:40.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:40.750+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:41.714 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:41.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:41.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:41.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:41.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:41.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:41.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:41.301+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:41.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:41.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:41.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:41.714+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:41.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:41.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:41.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:41.860+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:42.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:42.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:42.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:42.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:42.677 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:42.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:42.677 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:42.338+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:42.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:42.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:42.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:42.675+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:42.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:42.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:42.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:42.812+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:43.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:43.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:43.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:43.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:43.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:43.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:43.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:43.372+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:43.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:43.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:43.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:43.722+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:43.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:43.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:43.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:43.835+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:44.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:44.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:44.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:44.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:44.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:44.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:44.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:44.384+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:44.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:44.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:44.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:44.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:44.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:44.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:44.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:45.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:44.861+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:45.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:45.288+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48141 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:45.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:45.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:45.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:45.425+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:46.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:45.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:46.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:45.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:45.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:45.776+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:45.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:45.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:45.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:46.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:45.906+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:46.762 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:46.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:46.762 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:46.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:46.762 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:46.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:46.762 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:46.458+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:46.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:46.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:46.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:46.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:46.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:46.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:46.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:47.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:46.894+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:47.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:47.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:47.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:47.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:47.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:47.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:47.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:47.456+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:47.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:47.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:47.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:47.931+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:47.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:47.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:47.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:48.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:47.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:48.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:48.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:48.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:48.439+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:48.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:48.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:48.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:49.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:48.906+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:48.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:48.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:48.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:48.862+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:49.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:49.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:49.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:49.472+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:50.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:49.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:50.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:49.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:50.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:49.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:50.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:49.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:49.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:49.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:49.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:49.877+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:50.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:50.294+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48185 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:50.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:50.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:50.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:50.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:50.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:50.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:50.455+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:50.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:50.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:50.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:51.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:50.915+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:50.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:50.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:50.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:50.878+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:51.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:51.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:51.479+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:51.479+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:51.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:51.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:51.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:51.908+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:51.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:51.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:51.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:51.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:52.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:52.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:52.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:52.519+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:52.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:52.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:52.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:52.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:52.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:52.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:52.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:52.909+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:53.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:53.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:53.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:53.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:53.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:53.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:53.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:53.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:53.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:53.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:53.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:53.959+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:53.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:53.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:53.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:53.910+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:54.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:54.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:54.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:54.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:54.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:54.550+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:54.915 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:54.551+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:54.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:54.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:54.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:55.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:54.913+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:54.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:54.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:54.928+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:54.929+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:55.556 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:54:55.301+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48231 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:54:55.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:55.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:55.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:55.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:55.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:55.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:55.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:55.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:55.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:55.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:55.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:55.877+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:56.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:55.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:56.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:55.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:56.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:55.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:56.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:55.927+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:56.884 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:56.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:56.884 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:56.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:56.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:56.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:56.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:56.604+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:56.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:56.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:56.882+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:57.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:56.883+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:57.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:56.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:57.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:56.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:57.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:56.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:57.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:56.964+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:57.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:57.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:57.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:57.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:57.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:57.564+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:57.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:57.565+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:57.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:57.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:57.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:57.996+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:57.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:57.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:57.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:57.925+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:58.966 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:58.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:58.966 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:58.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:58.966 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:58.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:58.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:58.543+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:54:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:58.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:58.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:58.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:54:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:58.965+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:59.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:59.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:59.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:54:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:54:59.033+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:54:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:59.591+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:59.591+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:59.591+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:54:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:54:59.592+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:59.977+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:59.977+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:59.977+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:54:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:54:59.977+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:00.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:00.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:00.008+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:00.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:00.009+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:00.585 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:00.307+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48275 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:00.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:00.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:00.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:00.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:00.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:00.584+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:00.948 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:00.584+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:01.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:01.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:01.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:01.034+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:00.945+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:00.946+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:00.946+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:00.946+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:01.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:01.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:01.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:01.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:01.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:01.555+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:01.920 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:01.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:01.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:01.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:01.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:01.919+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:02.079+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:02.079+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:02.079+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:02.079+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:02.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:02.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:02.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:02.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:02.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:02.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:02.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:02.536+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:02.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:02.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:02.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:02.883+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:03.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:03.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:03.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:03.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:03.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:03.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:03.109+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:03.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:03.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:03.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:03.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:03.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:03.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:03.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:03.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:03.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:03.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:03.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:03.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:04.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:04.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:04.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:04.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:04.145+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:04.876 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:04.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:04.876 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:04.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:04.877 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:04.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:04.877 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:04.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:05.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:04.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:05.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:04.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:05.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:04.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:05.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:04.875+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:05.435 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:05.313+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48322 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:05.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:05.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:05.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:05.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:05.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:05.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:05.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:05.159+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:05.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:05.579+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:05.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:05.579+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:05.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:05.579+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:05.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:05.579+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:06.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:05.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:06.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:05.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:06.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:05.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:06.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:05.856+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:06.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:06.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:06.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:06.203+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:06.880 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:06.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:06.880 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:06.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:06.880 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:06.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:06.880 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:06.584+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:07.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:06.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:07.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:06.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:07.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:06.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:07.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:06.879+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:07.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:07.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:07.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:07.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:07.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:07.154+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:07.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:07.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:07.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:07.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:07.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:07.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:07.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:07.540+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:08.197 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:07.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:08.197 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:07.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:08.197 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:07.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:08.197 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:07.837+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:08.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:08.195+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:08.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:08.195+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:08.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:08.195+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:08.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:08.196+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:08.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:08.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:08.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:08.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:08.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:08.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:08.514+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:09.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:08.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:09.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:08.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:09.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:08.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:09.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:08.796+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:09.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:09.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:09.174+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:09.175+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:09.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:09.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:09.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:09.503+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:10.195 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:09.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:10.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:09.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:10.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:09.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:10.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:09.811+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:10.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:10.319+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48366 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:10.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:10.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:10.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:10.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:10.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:10.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:10.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:10.195+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:10.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:10.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:10.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:10.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:10.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:10.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:10.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:10.502+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:11.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:10.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:11.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:10.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:11.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:10.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:11.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:10.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:11.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:11.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:11.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:11.164+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:11.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:11.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:11.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:11.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:11.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:11.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:11.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:11.525+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:11.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:11.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:11.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:11.742+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:12.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:12.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:12.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:12.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:12.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:12.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:12.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:12.144+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:12.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:12.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:12.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:12.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:12.522+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:12.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:12.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:12.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:12.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:13.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:13.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:13.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:13.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:13.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:13.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:13.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:13.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:13.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:13.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:13.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:13.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:13.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:13.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:13.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:13.515+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:13.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:13.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:13.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:13.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:14.135+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:14.135+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:14.135+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:14.135+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:14.806 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:14.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:14.806 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:14.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:14.806 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:14.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:14.806 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:14.542+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:15.168 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:14.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:15.168 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:14.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:15.168 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:14.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:15.169 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:14.805+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:15.445 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:15.326+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48412 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:15.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:15.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:15.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:15.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:15.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:15.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:15.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:15.168+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:15.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:15.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:15.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:15.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:15.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:15.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:15.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:15.557+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:16.212 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:15.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:16.212 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:15.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:16.212 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:15.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:16.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:15.821+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:16.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:16.211+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:16.211+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:16.211+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:16.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:16.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:16.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:17.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:16.599+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:16.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:16.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:16.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:16.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:17.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:17.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:17.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:17.189+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:18.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:17.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:18.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:17.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:18.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:17.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:18.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:17.635+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:17.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:17.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:17.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:17.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:18.227+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:18.227+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:18.227+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:18.227+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:18.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:18.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:18.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:19.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:18.850+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:19.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:18.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:19.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:18.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:19.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:18.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:19.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:18.605+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:19.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:19.189+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:19.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:19.189+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:19.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:19.189+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:19.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:19.190+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:19.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:19.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:19.569+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:19.570+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:20.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:19.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:20.236 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:19.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:20.236 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:19.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:20.236 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:19.888+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:20.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:20.332+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48456 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:20.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:20.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:20.233+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:20.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:20.234+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:20.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:20.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:20.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:20.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:20.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:20.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:20.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:20.598+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:21.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:20.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:21.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:20.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:21.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:20.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:21.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:20.877+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:21.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:21.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:21.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:21.266+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:21.868 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:21.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:21.868 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:21.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:21.869 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:21.582+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:21.869 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:21.583+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:21.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:21.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:21.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:22.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:21.867+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:22.549 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:22.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:22.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:22.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:22.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:22.297+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:22.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:22.298+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:22.918 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:22.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:22.918 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:22.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:22.919 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:22.548+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:22.919 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:22.548+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:22.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:22.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:22.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:22.917+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:23.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:23.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:23.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:23.331+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:23.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:23.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:23.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:23.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:23.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:23.562+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:23.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:23.563+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:23.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:23.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:23.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:23.967+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:24.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:24.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:24.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:24.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:24.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:24.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:24.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:24.338+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:24.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:24.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:24.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:24.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:24.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:24.568+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:24.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:24.568+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:25.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:24.994+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:25.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:24.994+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:25.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:24.994+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:25.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:24.994+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:25.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:25.338+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48503 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:25.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:25.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:25.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:25.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:25.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:25.289+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:25.974 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:25.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:25.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:25.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:25.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:25.542+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:25.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:25.543+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:25.973+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:25.973+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:25.973+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:25.974+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:26.578 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:26.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:26.578 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:26.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:26.578 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:26.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:26.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:26.302+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:26.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:26.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:26.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:26.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:26.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:26.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:26.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:26.577+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:27.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:26.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:27.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:26.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:27.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:26.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:27.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:26.950+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:27.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:27.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:27.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:27.256+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:27.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:27.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:27.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:27.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:27.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:27.544+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:27.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:27.545+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:28.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:27.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:28.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:27.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:28.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:27.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:28.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:27.960+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:28.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:28.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:28.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:28.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:28.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:28.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:28.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:28.225+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:28.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:28.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:28.510+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:28.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:28.511+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:29.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:28.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:29.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:28.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:29.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:28.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:29.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:28.985+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:29.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:29.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:29.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:29.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:29.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:29.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:29.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:29.268+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:29.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:29.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:29.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:29.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:29.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:29.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:29.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:29.550+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:30.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:29.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:30.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:29.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:30.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:29.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:30.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:29.951+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:30.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:30.345+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48547 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:30.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:30.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:30.263+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:30.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:30.264+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:30.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:30.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:30.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:30.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:30.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:30.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:30.971 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:30.599+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:30.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:30.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:30.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:30.969+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:31.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:31.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:31.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:31.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:31.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:31.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:31.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:31.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:31.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:31.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:31.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:31.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:31.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:31.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:31.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:31.610+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:32.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:31.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:32.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:31.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:32.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:31.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:32.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:31.978+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:32.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:32.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:32.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:32.265+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:33.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:32.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:33.007 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:32.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:33.007 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:32.575+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:33.007 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:32.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:33.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:33.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:33.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:33.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:33.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:33.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:33.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:33.005+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:33.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:33.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:33.282+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:33.283+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:34.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:33.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:34.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:33.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:34.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:33.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:34.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:33.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:34.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:34.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:34.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:34.247+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:34.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:34.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:34.009+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:34.010+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:34.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:34.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:34.536+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:34.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:34.537+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:35.222+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:35.222+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:35.222+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:35.223+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:35.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:35.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:35.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:35.013+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:35.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:35.351+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48593 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:35.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:35.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:35.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:35.573+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:36.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:36.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:36.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:36.209+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:36.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:36.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:36.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:36.050+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:36.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:36.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:36.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:36.585+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:37.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:37.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:37.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:37.182+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:37.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:37.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:37.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:37.094+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:37.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:37.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:37.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:37.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:38.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:38.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:38.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:38.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:38.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:38.151+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:38.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:38.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:38.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:38.135+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:38.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:38.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:38.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:39.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:38.598+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:39.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:39.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:39.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:39.106+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:39.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:39.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:39.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:39.142+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:39.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:39.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:39.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:39.643+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:40.366 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:40.357+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48637 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:40.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:40.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:40.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:40.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:40.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:40.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:40.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:40.151+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:40.368 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:40.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:40.368 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:40.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:40.368 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:40.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:40.368 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:40.108+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:40.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:40.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:40.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:40.664+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:41.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:41.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:41.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:41.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:41.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:41.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:41.145+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:41.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:41.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:41.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:41.177+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:41.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:41.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:41.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:41.641+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:42.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:42.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:42.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:42.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:42.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:42.151+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:42.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:42.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:42.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:42.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:42.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:42.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:42.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:42.177+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:42.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:42.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:42.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:42.640+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:43.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:43.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:43.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:43.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:43.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:43.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:43.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:43.149+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:43.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:43.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:43.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:43.131+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:43.666+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:43.666+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:43.666+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:43.667+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:44.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:44.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:44.112+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:44.112+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:44.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:44.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:44.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:44.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:44.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:44.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:44.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:44.102+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:44.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:44.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:44.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:44.694+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:45.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:45.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:45.366 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:45.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:45.366 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:45.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:45.366 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:45.106+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:45.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:45.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:45.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:45.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:45.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:45.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:45.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:45.143+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:45.742 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:45.363+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48684 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:45.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:45.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:45.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:46.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:45.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:46.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:46.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:46.137+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:46.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:46.137+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:46.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:46.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:46.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:46.110+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:46.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:46.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:46.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:46.699+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:47.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:47.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:47.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:47.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:47.111+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:47.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:47.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:47.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:47.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:47.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:47.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:47.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:47.102+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:47.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:47.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:47.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:48.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:47.694+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:48.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:48.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:48.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:48.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:48.113+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:48.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:48.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:48.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:48.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:48.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:48.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:48.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:48.705+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:49.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:49.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:49.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:49.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:49.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:49.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:49.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:49.147+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:49.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:49.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:49.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:49.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:49.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:49.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:49.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:49.079+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:49.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:49.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:49.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:49.695+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:50.371 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:50.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:50.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:50.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:50.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:50.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:50.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:50.120+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:50.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:50.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:50.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:50.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:50.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:50.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:50.373 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:50.106+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:50.670 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:50.369+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48728 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:50.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:50.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:50.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:50.669+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:51.140+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:51.140+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:51.140+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:51.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:51.141+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:51.133+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:51.133+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:51.133+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:51.133+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:51.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:51.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:51.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:51.688+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:52.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:52.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:52.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:52.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:52.101+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:52.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:52.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:52.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:52.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:52.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:52.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:52.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:52.095+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:52.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:52.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:52.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:52.688+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:53.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:53.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:53.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:53.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:53.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:53.076+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:53.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:53.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:53.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:53.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:53.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:53.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:53.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:54.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:53.698+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:54.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:54.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:54.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:54.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:54.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:54.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:54.142+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:54.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:54.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:54.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:54.122+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:54.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:54.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:54.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:54.740+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:55.395 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:55:55.376+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48774 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:55:55.395 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:55.159+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:55.395 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:55.159+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:55.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:55.159+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:55.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:55.159+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:55.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:55.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:55.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:55.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:55.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:55.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:55.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:55.134+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:55.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:55.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:55.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:55.725+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:56.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:56.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:56.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:56.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:56.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:56.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:56.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:56.125+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:56.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:56.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:56.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:56.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:56.157+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:56.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:56.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:57.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:56.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:57.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:56.760+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:57.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:57.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:57.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:57.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:57.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:57.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:57.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:57.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:57.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:57.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:57.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:57.126+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:58.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:57.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:58.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:57.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:58.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:57.801+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:58.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:57.802+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:58.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:58.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:58.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:58.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:58.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:58.122+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:58.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:58.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:58.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:58.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:58.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:58.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:58.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:58.108+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:59.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:58.808+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:59.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:58.808+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:59.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:58.808+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:55:59.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:58.808+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:55:59.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:59.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:59.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:59.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:59.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:59.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:55:59.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:55:59.113+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:55:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:59.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:59.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:59.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:55:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:55:59.098+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:00.085 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:00.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:00.085 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:00.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:00.085 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:00.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:00.085 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:00.069+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:00.085 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:59.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:00.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:59.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:00.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:59.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:00.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:55:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:55:59.833+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:00.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:00.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:00.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:00.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:00.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:00.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:00.384 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:00.083+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:00.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:00.382+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48818 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:01.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:00.841+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:01.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:00.841+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:01.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:00.841+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:01.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:00.842+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:01.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:01.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:01.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:01.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:01.107+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:01.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:01.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:01.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:01.129+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:02.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:02.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:02.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:02.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:02.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:02.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:02.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:02.065+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:02.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:01.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:02.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:01.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:02.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:01.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:02.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:01.829+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:02.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:02.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:02.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:02.131+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:03.102 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:03.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:03.102 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:03.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:03.103 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:03.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:03.103 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:03.025+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:03.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:02.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:03.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:02.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:03.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:02.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:03.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:02.792+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:03.100+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:03.100+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:03.100+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:03.101+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:04.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:03.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:04.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:03.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:04.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:03.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:04.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:03.763+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:04.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:04.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:04.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:04.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:04.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:04.022+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:04.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:04.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:04.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:04.088+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:04.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:04.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:04.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:04.771+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:05.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:05.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:05.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:05.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:05.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:05.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:05.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:05.057+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:05.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:05.113+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:05.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:05.113+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:05.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:05.113+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:05.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:05.113+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:05.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:05.388+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48865 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:06.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:05.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:06.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:05.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:06.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:05.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:06.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:05.809+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:06.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:06.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:06.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:06.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:06.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:06.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:06.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:06.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:06.096+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:06.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:06.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:06.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:07.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:06.774+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:07.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:07.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:07.102+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:07.103+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:07.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:07.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:07.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:07.055+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:08.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:07.804+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:08.082 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:07.804+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:08.082 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:07.804+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:08.082 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:07.805+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:08.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:08.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:08.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:08.080+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:08.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:08.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:08.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:08.101+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:09.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:08.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:09.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:08.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:09.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:08.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:09.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:08.828+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:09.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:09.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:09.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:09.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:09.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:09.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:09.111+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:09.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:09.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:09.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:09.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:09.130+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:10.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:09.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:10.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:09.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:10.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:09.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:10.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:09.819+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:10.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:10.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:10.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:10.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:10.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:10.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:10.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:10.148+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:10.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:10.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:10.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:10.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:10.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:10.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:10.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:10.122+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:10.772 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:10.394+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48908 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:10.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:10.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:10.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:10.772+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:11.184+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:11.184+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:11.184+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:11.185+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:11.149+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:11.149+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:11.149+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:11.150+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:12.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:11.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:12.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:11.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:12.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:11.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:11.788+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:12.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:12.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:12.210+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:12.210+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:12.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:12.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:12.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:12.144+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:12.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:12.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:13.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:12.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:12.780+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:13.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:13.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:13.240+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:13.241+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:13.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:13.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:13.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:13.150+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:14.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:13.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:14.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:13.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:14.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:13.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:14.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:13.818+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:14.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:14.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:14.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:14.163+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:14.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:14.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:14.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:14.286+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:15.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:14.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:15.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:14.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:15.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:14.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:15.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:14.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:15.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:15.400+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48947 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:15.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:15.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:15.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:15.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:15.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:15.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:15.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:15.305+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:15.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:15.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:15.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:15.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:15.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:15.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:15.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:15.167+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:15.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:15.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:15.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:15.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:16.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:16.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:16.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:16.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:16.161+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:16.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:16.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:16.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:16.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:16.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:16.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:16.321+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:17.205 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:16.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:17.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:16.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:17.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:16.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:17.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:16.900+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:17.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:17.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:17.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:17.204+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:17.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:17.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:17.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:17.294+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:18.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:18.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:18.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:18.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:18.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:18.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:18.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:18.171+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:18.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:17.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:18.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:17.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:18.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:17.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:18.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:17.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:18.278+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:18.278+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:18.278+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:18.279+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:19.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:19.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:19.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:19.135+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:18.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:18.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:19.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:18.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:19.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:18.989+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:19.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:19.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:19.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:19.300+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:19.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:19.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:20.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:19.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:20.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:19.972+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:20.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:20.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:20.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:20.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:20.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:20.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:20.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:20.173+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:20.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:20.406+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 48990 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:20.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:20.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:20.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:20.331+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:20.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:20.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:20.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:20.947+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:21.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:21.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:21.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:21.176+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:21.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:21.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:21.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:21.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:21.293+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:21.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:21.294+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:22.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:22.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:22.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:22.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:22.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:22.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:22.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:22.221+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:22.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:21.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:22.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:21.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:22.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:21.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:22.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:21.991+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:22.280+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:22.280+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:22.280+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:22.281+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:23.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:23.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:23.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:23.268+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:23.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:23.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:23.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:23.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:23.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:23.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:23.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:23.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:23.332+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:24.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:24.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:24.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:24.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:24.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:24.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:24.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:24.313+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:24.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:24.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:24.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:24.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:25.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:25.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:25.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:25.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:25.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:25.412+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49037 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:25.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:25.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:25.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:25.366+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:25.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:25.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:25.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:25.333+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:25.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:25.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:25.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:25.989+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:26.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:26.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:26.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:26.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:26.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:26.323+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:26.797 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:26.324+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:26.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:26.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:26.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:26.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:26.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:26.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:26.797 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:26.298+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:26.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:26.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:26.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:26.973+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:27.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:27.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:27.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:27.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:27.349+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:27.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:27.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:27.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:27.308+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:28.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:28.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:28.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:28.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:28.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:28.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:28.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:28.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:28.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:28.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:28.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:28.275+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:28.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:28.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:28.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:28.314+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:29.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:29.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:29.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:29.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:29.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:29.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:29.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:29.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:29.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:29.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:29.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:29.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:29.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:29.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:29.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:29.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:29.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:29.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:29.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:29.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:29.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:29.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:29.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:29.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:30.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:30.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:30.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:30.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:30.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:30.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:30.231+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:30.231+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:30.231+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:30.231+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:30.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:30.418+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49081 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:30.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:30.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:30.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:04.750146+0000 (oldest deadline 2024-02-19T19:46:24.750146+0000) 2024-02-19T19:56:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:30.352+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:31.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:31.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:31.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:31.037+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:31.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:31.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:31.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:31.204+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:31.393+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:32.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:32.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:32.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:32.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:32.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:32.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:32.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:32.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:32.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:32.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:32.379 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:32.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:32.379 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:32.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:32.379 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:32.062+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:32.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:32.376+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:33.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:33.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:33.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:33.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:33.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:33.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:33.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:33.236+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:33.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:33.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:33.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:33.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:33.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:33.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:33.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:33.078+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:33.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:33.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:34.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:34.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:34.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:34.232+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:34.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:34.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:34.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:34.031+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:34.328+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:35.361 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:35.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:35.361 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:35.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:35.361 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:35.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:35.361 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:35.193+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:35.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:35.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:35.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:35.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:35.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:35.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:35.362 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:35.064+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:35.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:35.359+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:35.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:35.424+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49127 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:36.339+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:36.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:36.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:36.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:36.093+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:36.205+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:36.205+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:36.205+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:36.205+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:37.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:37.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:37.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:37.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:37.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:37.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:37.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:37.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:37.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:37.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:37.229+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:38.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:38.266+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:38.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:38.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:38.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:38.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:38.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:38.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:38.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:38.190+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:38.444 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:38.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:38.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:38.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:38.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:38.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:38.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:38.144+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:39.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:39.309+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:39.182+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:39.182+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:39.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:39.182+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:39.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:39.182+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:39.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:39.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:39.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:39.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:39.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:39.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:39.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:39.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:40.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:40.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:40.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:40.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:40.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:40.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:40.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:40.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:40.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:40.137+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:40.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:40.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:40.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:40.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:40.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:40.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:40.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:40.219+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:40.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:40.431+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49171 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:41.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:41.336+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:41.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:41.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:41.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:06.946235+0000 (oldest deadline 2024-02-19T19:46:26.946235+0000) 2024-02-19T19:56:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:41.118+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:41.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:41.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:41.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:41.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:41.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:41.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:41.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:41.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:42.151+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:42.296+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:42.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:42.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:42.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:46:09.515714+0000 (oldest deadline 2024-02-19T19:46:29.515714+0000) 2024-02-19T19:56:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:42.224+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:43.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:43.104+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:43.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:43.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:43.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:43.243+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:44.124+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:44.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:44.292+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:44.222+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:45.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:45.285+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:45.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:45.229+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:45.440 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:45.104+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:45.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:45.437+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49219 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:46.294+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:46.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:46.203+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:46.120+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:47.340 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:47.073+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:47.340 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:47.234+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:47.338+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:48.297+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:48.266+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:48.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:49.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:49.287+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:49.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:49.144+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:49.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:49.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:50.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:50.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:50.446 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:50.192+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:50.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:50.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:50.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:50.443+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49263 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:51.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:51.136+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:51.356+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:51.221+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:52.350+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:52.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:52.156+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:52.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:52.183+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:53.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:53.201+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:53.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:53.233+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:53.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:53.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:53.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:53.398+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:54.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:54.153+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:54.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:54.278+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:54.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:54.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:54.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:54.408+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:55.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:55.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:55.452 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:55.239+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:55.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:55.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:55.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:55.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:55.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:55.373+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:55.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:55.374+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:55.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:56:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:56:55.450+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49309 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:56:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:56.240+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:56.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:56.102+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:56.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:56.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:56.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:56.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:56.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:56.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:56.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:56.341+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:57.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:57.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:57.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:57.216+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:57.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:57.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:57.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:57.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:57.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:57.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:57.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:57.335+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:58.333 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:58.218+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:58.333 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:58.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:58.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:58.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:58.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:58.332+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:59.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:56:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:56:59.119+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:56:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:56:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:56:59.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:56:59.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:59.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:59.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:59.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:59.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:59.334+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:56:59.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:56:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:56:59.335+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:00.458 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:00.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:00.458 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:00.199+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:00.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:00.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:00.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:00.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:00.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:00.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:00.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:00.298+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:00.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:00.456+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49353 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:01.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:01.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:01.205+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:01.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:01.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:01.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:01.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:01.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:01.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:01.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:01.291+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:02.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:02.043+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:02.204+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:02.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:02.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:02.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:02.266+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:03.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:03.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:03.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:03.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:03.189+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:03.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:03.316+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:03.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:03.316+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:03.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:03.316+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:03.796 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:03.317+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:04.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:03.986+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:04.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:04.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:04.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:04.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:04.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:04.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:04.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:04.213+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:04.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:04.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:04.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:04.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:04.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:04.287+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:05.196 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:04.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:05.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:05.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:05.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:05.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:05.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:05.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:05.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:05.245+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:05.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:05.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:05.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:05.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:05.466 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:05.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:05.466 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:05.194+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:05.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:05.462+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49400 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:06.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:05.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:06.232 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:05.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:06.232 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:05.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:06.232 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:05.904+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:06.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:06.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:06.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:06.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:06.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:06.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:06.253+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:06.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:06.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:06.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:06.230+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:07.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:06.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:07.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:06.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:07.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:06.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:07.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:06.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:07.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:07.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:07.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:07.268+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:07.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:07.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:07.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:07.266+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:08.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:07.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:08.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:07.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:08.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:07.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:08.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:07.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:08.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:08.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:08.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:08.276+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:08.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:08.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:08.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:08.302+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:08.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:08.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:08.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:08.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:09.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:09.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:09.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:09.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:09.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:09.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:09.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:09.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:09.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:09.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:09.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:09.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:09.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:09.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:09.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:09.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:10.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:09.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:10.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:09.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:10.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:09.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:10.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:09.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:10.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:10.469+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49442 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:10.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:10.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:10.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:10.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:10.309+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:10.310+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:10.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:10.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:10.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:10.249+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:11.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:10.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:11.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:10.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:11.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:10.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:11.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:10.920+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:11.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:11.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:11.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:11.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:11.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:11.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:11.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:11.276+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:12.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:11.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:12.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:11.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:12.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:11.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:12.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:11.873+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:12.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:12.325+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:12.325+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:12.326+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:12.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:12.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:12.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:12.251+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:13.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:12.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:13.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:12.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:13.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:12.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:13.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:12.893+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:13.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:13.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:13.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:13.289+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:13.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:13.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:13.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:13.219+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:14.177 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:13.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:14.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:13.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:14.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:13.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:14.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:13.844+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:14.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:14.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:14.338+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:14.339+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:14.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:14.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:14.176+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:14.176+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:15.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:14.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:15.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:14.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:15.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:14.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:15.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:14.877+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:15.477 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:15.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:15.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:15.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:15.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:15.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:15.478 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:15.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:15.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:15.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:15.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:15.151+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:15.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:15.475+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49483 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:16.137 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:15.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:16.138 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:15.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:16.138 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:15.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:16.138 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:15.845+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:16.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:16.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:16.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:16.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:16.323+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:16.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:16.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:16.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:16.136+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:17.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:16.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:17.170 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:16.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:17.170 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:16.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:17.170 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:16.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:17.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:17.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:17.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:17.333+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:17.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:17.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:17.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:17.168+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:18.200 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:17.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:18.200 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:17.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:18.200 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:17.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:18.200 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:17.943+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:18.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:18.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:18.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:18.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:18.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:18.198+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:18.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:18.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:18.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:18.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:18.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:18.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:18.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:18.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:18.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:19.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:19.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:19.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:19.181+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:19.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:19.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:19.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:19.313+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:20.197 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:19.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:20.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:19.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:20.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:19.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:20.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:19.901+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:20.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:20.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:20.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:20.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:20.485 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:20.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:20.485 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:20.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:20.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:20.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:20.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:20.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:20.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:20.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:20.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:20.196+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:20.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:20.482+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49521 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:21.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:20.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:21.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:20.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:21.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:20.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:21.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:20.926+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:21.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:21.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:21.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:21.396+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:21.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:21.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:21.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:21.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:22.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:21.970+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:22.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:21.970+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:22.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:21.970+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:22.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:21.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:22.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:22.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:22.348+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:22.349+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:22.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:22.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:22.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:22.270+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:22.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:22.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:22.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:22.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:23.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:23.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:23.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:23.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:23.225+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:23.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:23.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:23.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:23.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:23.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:24.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:24.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:24.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:24.286+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:24.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:24.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:24.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:24.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:24.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:24.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:24.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:24.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:24.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:24.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:24.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:24.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:24.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:25.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:25.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:25.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:25.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:25.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:25.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:25.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:25.314+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:25.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:25.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:25.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:25.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:25.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:25.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:25.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:25.078+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:25.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:25.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:25.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:25.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:25.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:25.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:25.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:25.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:25.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:25.489+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49561 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:26.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:26.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:26.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:26.035+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:26.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:26.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:26.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:26.235+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:26.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:26.235+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:26.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:26.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:26.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:26.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:26.315+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:27.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:27.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:27.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:27.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:27.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:27.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:27.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:27.207+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:27.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:27.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:27.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:27.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:27.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:27.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:28.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:28.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:28.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:28.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:28.324+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:28.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:28.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:28.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:28.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:28.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:28.089+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:28.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:28.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:28.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:28.200+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:29.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:29.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:29.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:29.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:29.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:29.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:29.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:29.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:29.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:29.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:29.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:29.073+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:29.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:29.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:29.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:29.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:29.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:29.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:29.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:29.178+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:30.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:30.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:30.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:30.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:30.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:30.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:30.337 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:30.045+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:30.337 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:30.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:30.337 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:30.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:30.337 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:30.152+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:30.337 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:30.152+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:30.613 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:30.496+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49599 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:30.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:30.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:30.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:30.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:30.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:30.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:30.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:30.336+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:31.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:31.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:31.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:31.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:31.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:31.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:31.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:31.179+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:31.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:31.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:31.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:31.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:32.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:32.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:32.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:32.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:32.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:32.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:32.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:32.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:32.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:32.142+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:32.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:32.142+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:32.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:32.142+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:32.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:32.142+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:32.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:32.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:32.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:32.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:32.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:32.403+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:33.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:33.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:33.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:33.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:33.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:33.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:33.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:33.091+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:33.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:33.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:33.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:33.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:33.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:33.167+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:33.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:33.168+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:33.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:33.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:33.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:33.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:33.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:33.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:34.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:34.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:34.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:34.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:34.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:34.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:34.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:34.107+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:34.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:34.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:34.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:34.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:34.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:34.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:34.427 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:34.173+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:34.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:34.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:34.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:34.424+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:35.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:35.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:35.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:35.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:35.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:35.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:35.462 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:35.140+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:35.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:35.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:35.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:35.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:35.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:35.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:35.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:35.062+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:35.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:35.501+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49639 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:35.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:35.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:35.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:35.460+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:36.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:36.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:36.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:36.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:36.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:36.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:36.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:36.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:36.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:36.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:36.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:36.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:36.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:36.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:36.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:36.153+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:36.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:36.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:36.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:36.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:36.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:37.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:37.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:37.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:37.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:37.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:37.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:37.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:37.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:37.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:37.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:37.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:37.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:37.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:37.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:37.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:37.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:37.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:37.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:37.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:37.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:37.536+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:38.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:38.066+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:38.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:38.066+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:38.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:38.066+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:38.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:38.066+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:38.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:38.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:38.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:38.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:38.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:38.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:38.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:38.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:38.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:38.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:38.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:38.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:38.523+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:39.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:39.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:39.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:39.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:39.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:39.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:39.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:39.031+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:39.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:39.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:39.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:39.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:39.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:39.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:39.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:39.022+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:39.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:39.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:39.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:39.477+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:40.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:40.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:40.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:40.043+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:40.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:39.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:40.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:39.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:40.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:39.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:40.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:39.992+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:40.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:40.507+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49677 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:40.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:40.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:40.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:40.438+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:41.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:41.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:41.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:41.082+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:40.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:40.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:40.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:40.984+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:41.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:41.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:41.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:41.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:41.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:41.438+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:42.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:42.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:42.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:42.064+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:41.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:41.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:41.975+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:41.976+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:42.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:42.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:42.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:42.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:42.397+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:43.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:43.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:43.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:43.084+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:42.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:42.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:42.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:42.942+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:43.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:43.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:43.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:43.350+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:43.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:44.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:43.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:44.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:43.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:44.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:43.944+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:44.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:44.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:44.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:44.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:44.034+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:44.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:44.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:44.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:44.366+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:45.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:45.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:45.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:45.061+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:44.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:44.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:44.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:44.920+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:45.634 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:45.513+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49717 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:45.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:45.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:45.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:45.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:45.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:45.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:45.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:45.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:45.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:45.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:45.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:45.923+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:46.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:46.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:46.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:46.058+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:46.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:46.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:46.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:46.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:47.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:47.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:47.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:47.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:46.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:46.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:46.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:46.954+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:47.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:47.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:47.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:47.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:47.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:47.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:47.962+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:47.962+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:48.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:48.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:48.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:48.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:48.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:48.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:48.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:48.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:48.988+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:49.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:48.988+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:49.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:48.988+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:49.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:48.988+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:49.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:49.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:49.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:49.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:49.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:49.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:49.351+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:49.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:50.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:49.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:50.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:49.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:50.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:49.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:50.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:49.942+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:50.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:50.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:50.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:50.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:50.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:50.519+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49755 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:50.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:50.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:50.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:50.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:51.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:51.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:51.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:51.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:50.971+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:50.971+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:50.971+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:50.971+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:51.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:51.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:51.360+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:51.361+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:52.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:52.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:52.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:52.056+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:51.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:51.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:52.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:51.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:52.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:51.951+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:52.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:52.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:52.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:52.397+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:53.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:53.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:53.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:53.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:52.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:52.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:52.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:52.945+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:53.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:53.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:53.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:53.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:53.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:54.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:54.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:54.095+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:54.096+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:53.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:54.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:53.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:54.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:53.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:54.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:53.899+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:54.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:54.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:54.394+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:54.395+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:55.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:54.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:55.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:54.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:55.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:54.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:55.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:54.856+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:55.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:55.361+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:55.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:55.361+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:55.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:55.361+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:55.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:55.362+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:55.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:55.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:55.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:55.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:55.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:55.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:55.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:55.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:55.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:57:55.525+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49794 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:57:56.133 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:55.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:56.133 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:55.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:56.133 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:55.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:56.133 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:55.854+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:56.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:56.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:56.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:56.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:56.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:56.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:56.414 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:56.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:56.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:56.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:56.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:56.412+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:57.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:57.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:57.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:57.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:57.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:56.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:57.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:56.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:57.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:56.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:57.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:56.892+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:57.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:57.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:57.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:57.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:58.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:58.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:58.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:58.087+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:57.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:57.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:57.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:57.868+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:58.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:58.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:58.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:58.415+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:59.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:59.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:59.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:57:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:57:59.063+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:57:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:58.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:58.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:58.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:57:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:58.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:57:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:59.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:59.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:59.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:57:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:57:59.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:00.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:00.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:00.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:00.031+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:00.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:59.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:00.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:59.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:00.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:59.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:00.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:57:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:57:59.845+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:00.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:00.532+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49833 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:00.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:00.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:00.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:00.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:00.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:00.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:00.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:00.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:00.852+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:01.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:01.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:01.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:01.049+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:01.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:01.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:01.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:01.490+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:01.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:01.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:01.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:01.864+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:02.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:02.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:02.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:02.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:02.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:02.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:02.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:02.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:02.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:02.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:03.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:03.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:03.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:03.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:02.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:02.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:02.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:02.904+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:03.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:03.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:03.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:03.423+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:03.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:03.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:03.900+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:03.900+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:04.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:04.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:04.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:04.017+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:04.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:04.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:04.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:04.430+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:05.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:05.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:05.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:05.045+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:04.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:04.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:04.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:04.887+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:05.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:05.538+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49872 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:05.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:05.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:05.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:05.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:06.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:06.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:06.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:06.035+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:06.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:05.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:06.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:05.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:06.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:05.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:06.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:05.869+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:06.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:06.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:06.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:06.498+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:06.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:06.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:07.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:06.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:07.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:06.854+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:07.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:07.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:07.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:07.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:07.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:07.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:07.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:07.473+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:08.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:08.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:08.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:08.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:07.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:07.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:07.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:08.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:07.903+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:08.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:08.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:08.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:08.454+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:08.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:08.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:08.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:08.875+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:09.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:09.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:09.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:09.085+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:09.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:09.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:09.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:09.474+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:10.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:09.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:10.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:09.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:10.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:09.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:10.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:09.846+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:10.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:10.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:10.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:10.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:10.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:10.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:10.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:10.069+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:10.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:10.544+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49911 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:10.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:10.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:10.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:10.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:11.084 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:10.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:11.085 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:10.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:11.085 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:10.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:11.085 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:10.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:11.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:11.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:11.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:11.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:11.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:11.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:11.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:11.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:11.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:11.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:11.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:11.460+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:11.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:12.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:11.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:12.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:11.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:12.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:11.813+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:12.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:12.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:12.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:12.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:12.060+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:12.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:12.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:12.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:12.440+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:13.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:13.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:13.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:13.046+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:12.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:12.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:12.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:12.854+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:13.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:13.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:13.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:13.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:13.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:13.441+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:13.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:14.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:13.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:14.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:13.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:14.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:13.903+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:14.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:14.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:14.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:14.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:14.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:14.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:14.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:14.454+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:14.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:14.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:14.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:14.995+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:15.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:14.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:15.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:14.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:15.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:14.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:15.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:14.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:15.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:15.550+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49950 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:15.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:15.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:15.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:15.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:15.475+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:16.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:16.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:16.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:16.025+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:15.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:15.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:15.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:16.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:15.851+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:16.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:16.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:16.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:16.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:16.497+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:16.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:16.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:16.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:16.812+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:16.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:16.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:16.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:16.980+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:17.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:17.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:17.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:17.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:17.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:17.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:17.525+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:17.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:17.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:17.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:17.969+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:17.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:17.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:17.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:17.839+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:18.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:18.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:18.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:18.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:18.494+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:18.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:18.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:18.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:18.973+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:18.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:18.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:18.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:18.794+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:19.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:19.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:19.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:19.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:19.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:19.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:19.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:19.462+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:19.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:20.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:19.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:20.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:19.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:20.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:19.785+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:20.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:19.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:20.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:19.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:20.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:19.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:20.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:19.928+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:20.754 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:20.556+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 49989 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:20.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:20.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:20.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:20.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:20.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:20.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:20.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:20.500+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:21.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:20.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:21.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:20.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:21.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:20.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:21.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:20.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:20.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:20.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:20.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:20.752+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:21.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:21.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:21.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:21.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:21.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:21.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:21.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:21.461+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:21.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:21.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:21.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:21.752+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:21.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:22.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:21.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:22.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:21.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:22.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:21.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:22.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:22.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:22.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:22.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:22.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:22.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:22.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:22.454+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:23.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:22.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:22.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:22.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:22.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:22.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:22.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:22.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:23.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:22.717+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:23.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:23.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:23.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:23.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:23.690+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:23.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:23.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:23.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:23.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:23.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:23.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:23.496+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:23.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:23.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:23.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:23.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:24.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:24.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:24.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:24.493+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:24.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:24.494+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:24.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:24.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:24.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:24.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:24.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:24.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:24.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:24.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:24.994+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:25.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:25.562+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50028 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:25.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:25.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:25.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:25.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:25.461+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:25.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:25.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:25.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:25.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:25.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:25.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:26.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:26.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:26.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:26.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:26.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:26.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:26.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:26.647+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:26.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:26.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:26.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:26.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:26.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:26.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:26.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:26.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:26.996+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:27.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:27.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:27.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:27.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:27.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:27.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:27.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:27.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:27.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:27.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:27.606+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:27.948+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:27.948+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:27.948+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:27.949+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:28.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:28.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:28.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:28.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:28.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:28.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:28.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:28.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:28.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:28.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:28.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:28.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:28.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:28.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:28.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:28.943+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:29.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:29.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:29.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:29.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:29.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:29.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:29.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:29.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:29.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:29.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:30.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:29.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:30.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:29.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:30.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:29.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:30.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:30.568+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50067 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:30.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:30.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:30.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:30.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:30.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:30.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:30.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:30.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:30.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:30.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:30.487+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:30.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:30.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:30.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:30.987+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:31.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:31.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:31.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:31.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:31.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:31.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:31.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:31.440+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:31.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:31.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:31.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:31.996+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:32.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:32.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:32.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:32.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:32.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:32.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:32.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:32.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:32.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:32.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:32.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:32.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:32.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:33.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:33.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:33.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:33.014+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:33.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:33.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:33.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:33.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:33.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:33.542+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:33.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:33.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:33.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:33.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:33.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:34.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:34.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:34.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:34.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:34.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:34.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:34.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:34.455+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:34.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:34.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:34.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:34.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:34.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:34.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:35.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:35.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:35.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:35.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:35.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:35.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:35.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:35.050+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:35.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:35.574+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50106 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:35.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:35.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:35.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:35.514+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:35.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:35.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:35.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:35.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:35.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:35.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:35.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:35.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:36.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:36.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:36.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:36.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:36.017+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:36.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:36.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:36.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:36.495+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:36.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:36.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:36.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:36.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:37.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:37.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:37.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:37.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:37.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:37.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:37.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:37.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:37.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:37.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:37.481+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:37.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:37.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:37.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:37.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:37.457+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:38.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:38.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:38.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:38.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:38.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:38.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:38.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:38.046+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:38.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:38.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:38.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:38.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:38.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:38.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:38.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:38.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:38.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:39.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:39.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:39.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:39.000+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:39.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:39.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:39.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:39.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:39.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:39.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:39.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:39.477+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:39.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:39.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:39.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:39.985+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:40.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:40.581+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50145 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:40.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:40.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:40.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:40.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:40.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:40.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:40.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:40.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:40.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:40.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:40.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:41.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:41.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:41.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:41.034+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:41.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:41.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:41.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:41.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:41.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:41.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:41.496+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:41.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:41.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:41.526+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:41.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:41.527+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:42.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:42.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:42.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:42.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:42.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:42.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:42.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:42.062+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:42.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:42.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:42.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:42.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:42.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:42.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:42.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:42.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:42.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:42.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:42.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:42.566+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:43.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:43.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:43.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:43.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:43.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:43.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:43.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:43.100+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:43.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:43.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:43.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:43.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:43.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:43.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:43.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:43.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:43.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:43.520+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:44.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:44.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:44.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:44.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:44.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:44.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:44.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:44.116+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:44.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:44.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:44.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:44.526+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:44.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:44.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:44.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:44.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:45.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:45.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:45.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:45.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:45.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:45.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:45.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:45.092+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:45.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:45.587+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50184 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:45.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:45.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:45.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:45.540+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:45.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:45.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:45.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:45.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:46.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:46.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:46.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:46.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:46.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:46.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:46.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:46.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:46.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:46.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:46.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:46.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:46.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:46.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:46.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:46.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:46.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:47.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:47.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:47.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:47.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:47.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:47.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:47.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:47.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:47.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:47.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:47.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:47.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:48.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:48.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:48.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:48.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:48.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:48.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:48.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:48.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:48.498+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:48.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:48.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:48.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:48.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:49.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:49.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:49.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:49.025+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:49.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:49.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:49.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:49.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:49.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:49.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:49.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:49.557+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:50.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:50.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:50.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:50.033+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:50.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:50.593+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50223 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:50.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:50.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:50.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:50.549+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:50.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:50.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:50.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:50.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:50.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:50.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:50.470+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:51.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:51.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:51.509 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:51.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:51.509 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:51.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:51.509 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:51.046+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:51.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:51.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:51.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:51.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:51.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:51.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:51.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:51.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:51.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:52.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:52.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:52.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:52.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:52.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:52.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:52.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:52.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:52.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:52.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:52.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:52.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:52.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:52.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:52.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:52.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:52.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:53.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:53.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:53.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:53.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:53.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:53.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:53.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:53.070+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:53.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:53.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:53.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:53.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:53.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:53.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:53.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:53.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:53.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:53.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:53.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:54.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:54.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:54.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:54.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:54.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:54.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:54.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:54.058+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:54.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:54.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:54.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:54.494+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:54.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:54.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:54.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:54.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:54.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:54.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:55.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:55.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:55.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:55.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:55.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:55.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:55.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:55.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:55.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:58:55.599+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50262 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:58:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:55.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:55.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:55.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:55.522+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:55.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:55.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:55.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:55.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:55.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:56.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:56.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:56.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:56.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:56.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:56.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:56.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:56.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:56.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:56.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:56.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:56.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:56.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:56.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:56.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:56.526+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:57.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:57.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:57.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:57.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:57.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:57.058+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:57.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:57.058+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:57.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:57.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:57.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:57.557+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:57.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:57.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:57.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:57.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:57.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:58.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:58.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:58.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:58.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:58.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:58.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:58.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:58.071+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:58.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:58.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:58.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:58.489+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:58.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:58.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:58.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:58.571+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:59.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:59.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:59.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:58:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:58:59.043+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:58:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:59.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:59.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:59.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:58:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:58:59.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:58:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:59.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:59.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:59.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:58:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:58:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:58:59.512+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:00.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:00.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:00.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:00.024+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:01.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:00.605+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50301 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:00.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:00.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:00.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:01.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:00.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:00.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:01.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:00.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:01.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:00.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:01.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:00.548+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:01.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:01.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:01.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:01.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:01.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:01.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:01.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:01.055+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:01.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:01.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:01.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:01.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:01.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:01.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:01.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:01.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:02.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:02.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:02.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:02.011+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:02.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:02.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:02.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:02.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:02.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:02.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:02.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:02.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:03.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:03.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:03.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:03.005+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:03.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:03.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:03.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:03.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:03.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:03.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:03.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:03.536+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:04.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:04.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:04.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:04.026+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:04.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:04.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:04.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:04.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:04.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:04.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:04.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:04.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:05.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:05.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:05.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:05.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:05.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:05.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:05.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:05.055+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:05.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:05.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:05.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:05.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:05.471+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:05.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:05.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:05.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:05.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:05.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:05.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:05.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:05.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:05.735 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:05.611+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50340 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:06.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:06.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:06.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:06.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:06.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:06.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:06.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:06.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:06.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:06.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:06.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:06.481+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:06.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:06.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:06.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:06.521+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:07.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:07.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:07.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:07.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:07.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:07.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:07.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:07.095+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:07.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:07.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:07.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:07.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:07.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:07.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:07.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:07.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:07.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:07.495+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:08.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:08.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:08.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:08.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:08.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:08.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:08.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:08.092+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:08.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:08.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:08.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:08.438+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:08.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:08.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:08.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:08.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:08.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:09.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:09.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:09.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:09.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:09.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:09.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:09.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:09.124+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:09.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:09.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:09.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:09.424+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:09.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:09.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:09.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:09.398+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:09.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:09.399+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:10.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:10.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:10.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:10.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:10.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:10.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:10.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:10.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:10.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:10.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:10.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:10.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:10.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:10.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:10.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:10.135+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:10.739 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:10.618+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50379 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:10.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:10.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:10.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:10.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:10.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:10.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:10.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:10.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:11.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:11.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:11.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:11.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:11.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:11.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:11.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:11.344+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:11.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:11.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:11.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:11.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:11.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:11.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:11.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:11.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:11.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:11.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:11.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:11.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:11.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:12.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:12.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:12.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:12.382+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:12.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:12.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:12.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:12.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:12.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:12.369+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:12.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:12.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:12.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:12.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:12.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:12.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:12.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:12.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:13.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:13.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:13.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:13.389+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:13.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:13.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:13.345+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:13.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:13.346+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:13.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:13.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:13.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:13.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:13.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:13.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:13.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:13.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:14.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:14.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:14.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:14.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:14.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:14.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:14.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:14.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:14.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:14.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:14.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:14.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:14.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:15.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:15.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:15.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:15.297+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:15.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:15.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:15.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:15.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:15.124+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:15.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:15.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:15.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:15.366+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:16.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:15.624+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50418 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:16.412 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:16.268+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:16.412 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:16.268+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:16.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:16.268+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:16.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:16.268+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:16.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:16.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:16.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:16.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:16.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:16.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:16.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:16.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:16.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:16.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:16.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:16.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:16.411+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:17.438 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:17.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:17.438 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:17.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:17.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:17.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:17.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:17.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:17.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:17.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:17.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:17.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:17.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:17.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:17.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:17.135+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:17.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:17.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:17.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:17.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:18.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:18.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:18.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:18.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:18.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:18.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:18.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:18.268+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:18.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:18.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:18.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:18.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:18.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:18.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:18.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:18.170+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:18.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:18.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:18.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:18.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:18.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:18.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:18.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:18.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:19.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:19.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:19.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:19.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:19.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:19.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:19.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:19.234+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:19.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:19.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:19.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:19.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:19.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:19.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:19.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:19.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:19.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:19.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:19.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:19.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:20.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:20.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:20.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:20.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:20.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:20.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:20.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:20.223+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:20.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:20.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:20.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:20.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:20.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:20.176+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:20.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:20.177+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:20.750 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:20.630+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50457 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:20.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:20.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:20.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:20.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:20.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:20.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:20.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:20.496+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:21.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:21.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:21.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:21.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:21.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:21.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:21.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:21.223+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:21.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:21.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:21.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:21.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:21.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:21.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:21.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:21.160+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:21.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:21.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:21.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:21.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:21.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:21.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:21.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:21.494+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:22.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:22.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:22.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:22.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:22.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:22.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:22.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:22.249+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:22.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:22.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:22.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:22.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:22.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:22.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:22.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:22.124+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:22.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:22.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:22.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:22.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:22.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:22.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:22.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:22.464+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:23.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:23.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:23.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:23.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:23.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:23.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:23.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:23.239+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:23.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:23.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:23.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:23.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:23.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:23.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:23.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:23.160+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:23.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:23.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:23.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:23.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:23.424+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:24.462 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:24.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:24.462 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:24.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:24.462 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:24.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:24.462 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:24.277+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:24.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:24.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:24.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:24.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:24.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:24.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:24.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:24.167+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:24.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:24.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:24.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:24.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:24.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:24.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:25.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:25.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:25.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:25.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:25.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:25.246+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:25.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:25.246+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:25.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:25.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:25.457 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:25.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:25.457 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:25.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:25.457 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:25.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:25.757 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:25.636+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50496 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:25.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:25.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:25.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:25.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:25.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:25.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:25.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:25.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:26.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:26.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:26.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:26.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:26.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:26.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:26.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:26.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:26.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:26.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:26.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:26.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:26.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:26.219+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:26.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:26.220+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:26.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:26.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:26.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:26.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:27.460 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:27.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:27.461 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:27.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:27.461 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:27.257+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:27.461 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:27.258+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:27.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:27.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:27.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:27.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:27.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:27.180+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:27.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:27.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:27.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:27.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:27.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:27.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:27.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:28.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:28.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:28.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:28.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:28.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:28.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:28.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:28.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:28.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:28.289+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:28.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:28.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:28.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:28.457+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:29.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:29.327+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:29.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:29.327+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:29.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:29.327+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:29.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:29.327+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:29.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:29.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:29.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:29.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:29.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:29.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:29.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:29.273+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:29.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:29.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:29.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:29.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:29.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:29.455+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:29.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:29.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:30.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:30.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:30.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:30.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:30.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:30.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:30.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:30.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:30.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:30.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:30.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:30.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:30.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:30.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:30.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:30.248+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:30.763 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:30.642+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50535 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:30.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:30.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:30.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:30.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:30.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:30.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:30.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:30.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:31.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:31.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:31.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:31.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:31.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:31.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:31.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:31.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:31.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:31.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:31.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:31.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:31.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:31.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:31.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:31.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:31.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:31.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:31.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:31.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:31.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:31.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:31.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:32.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:32.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:32.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:32.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:32.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:32.409+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:32.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:32.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:32.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:32.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:32.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:32.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:32.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:32.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:33.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:33.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:33.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:33.403+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:33.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:33.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:33.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:33.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:33.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:33.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:33.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:33.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:33.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:33.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:33.397+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:34.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:34.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:34.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:34.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:34.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:34.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:34.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:34.403+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:34.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:34.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:34.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:34.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:34.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:34.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:34.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:34.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:34.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:34.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:34.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:34.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:34.443+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:35.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:35.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:35.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:35.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:35.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:35.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:35.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:35.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:35.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:35.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:35.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:35.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:35.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:35.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:35.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:35.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:35.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:35.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:36.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:35.648+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50574 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:36.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:36.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:36.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:36.439+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:36.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:36.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:36.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:36.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:36.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:36.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:36.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:36.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:36.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:36.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:36.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:36.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:36.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:37.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:37.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:37.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:37.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:37.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:37.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:37.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:37.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:37.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:37.420+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:37.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:37.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:37.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:37.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:37.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:37.258+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:37.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:37.259+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:38.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:38.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:38.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:38.256+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:38.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:38.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:38.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:38.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:38.468+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:38.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:38.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:38.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:38.431+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:39.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:39.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:39.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:39.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:39.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:39.469+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:39.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:39.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:39.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:39.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:39.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:39.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:39.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:39.243+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:39.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:39.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:39.501+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:39.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:40.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:40.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:40.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:40.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:40.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:40.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:40.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:40.269+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:40.774 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:40.655+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50613 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:40.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:40.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:40.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:40.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:40.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:40.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:40.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:40.534+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:40.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:40.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:40.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:40.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:40.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:40.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:40.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:40.514+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:41.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:41.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:41.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:41.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:41.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:41.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:41.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:41.257+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:41.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:41.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:41.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:41.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:41.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:41.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:41.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:41.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:41.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:41.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:42.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:42.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:42.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:42.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:42.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:42.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:42.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:42.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:42.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:42.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:42.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:42.479+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:42.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:42.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:42.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:42.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:42.545 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:42.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:43.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:43.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:43.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:43.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:43.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:43.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:43.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:43.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:43.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:43.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:43.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:43.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:43.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:43.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:43.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:43.245+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:43.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:43.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:43.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:43.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:43.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:43.506+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:44.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:44.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:44.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:44.445+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:44.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:44.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:44.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:44.253+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:45.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:44.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:45.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:44.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:45.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:44.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:45.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:44.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:45.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:45.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:45.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:45.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:45.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:45.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:45.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:45.401+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:45.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:45.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:45.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:45.224+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:46.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:45.661+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50652 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:45.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:45.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:45.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:45.572+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:46.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:46.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:46.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:46.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:46.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:46.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:46.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:46.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:46.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:46.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:46.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:46.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:46.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:46.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:46.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:46.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:46.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:47.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:47.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:47.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:47.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:47.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:47.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:47.237+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:47.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:47.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:47.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:47.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:47.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:47.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:47.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:47.385+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:47.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:47.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:47.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:47.625+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:48.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:48.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:48.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:48.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:48.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:48.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:48.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:48.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:48.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:48.351+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:48.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:48.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:48.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:48.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:49.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:49.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:49.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:49.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:49.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:49.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:49.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:49.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:49.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:49.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:49.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:49.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:49.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:49.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:49.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:49.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:50.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:49.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:50.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:49.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:49.695+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:49.696+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:50.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:50.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:50.309+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:50.310+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:50.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:50.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:50.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:50.382+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:51.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:50.667+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50691 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:50.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:50.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:50.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:50.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:51.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:51.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:51.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:51.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:51.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:51.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:51.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:51.329+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:51.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:51.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:51.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:51.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:51.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:51.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:51.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:51.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:51.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:52.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:51.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:52.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:51.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:52.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:51.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:52.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:52.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:52.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:52.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:52.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:52.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:52.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:52.385+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:52.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:52.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:52.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:52.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:52.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:52.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:52.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:52.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:52.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:52.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:52.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:53.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:52.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:53.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:53.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:53.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:53.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:53.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:53.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:53.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:53.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:53.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:53.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:53.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:53.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:53.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:53.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:53.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:53.390+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:54.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:53.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:53.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:53.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:53.697+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:54.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:54.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:54.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:54.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:54.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:54.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:54.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:54.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:54.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:54.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:54.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:54.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:54.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:54.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:54.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:54.366+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:54.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:54.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:54.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:54.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:55.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:55.413+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:55.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:55.413+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:55.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:55.413+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:55.676 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:55.414+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:55.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:55.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:55.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:55.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:55.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:55.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:55.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:55.389+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:56.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T19:59:55.674+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50730 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T19:59:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:55.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:55.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:55.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:55.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:56.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:56.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:56.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:56.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:56.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:56.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:56.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:56.389+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:56.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:56.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:56.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:56.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:56.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:56.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:56.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:56.376+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:57.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:56.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:56.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:56.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:56.667+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:57.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:57.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:57.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:57.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:57.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:57.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:57.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:57.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:57.409+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:57.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:57.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:57.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:57.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:57.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:57.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:57.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:57.388+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:58.672 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:58.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:58.672 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:58.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:58.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:58.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:58.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:58.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:58.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:58.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:58.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:58.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:58.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:58.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:58.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:58.436+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:58.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:58.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:58.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:58.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:59.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:59.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:59.638+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T19:59:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T19:59:59.639+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T19:59:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:59.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:59.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:59.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:59.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T19:59:59.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T19:59:59.429+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T19:59:59.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:59.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:59.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:59.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:59.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:59.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T19:59:59.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 19:59:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T19:59:59.457+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:00.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:00.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:00.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:00.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:00.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:00.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:00.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:00.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:00.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:00.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:00.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:00.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:00.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:00.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:00.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:00.408+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:00.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:00.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:00.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:00.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:00.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:00.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:00.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:00.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:01.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:00.680+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50769 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:01.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:01.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:01.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:01.635+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:01.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:01.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:01.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:01.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:01.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:01.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:01.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:01.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:01.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:01.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:01.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:01.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:01.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:02.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:02.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:02.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:02.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:02.685+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:02.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:02.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:02.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:02.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:02.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:02.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:02.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:02.440+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:02.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:02.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:02.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:02.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:02.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:02.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:02.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:02.458+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:03.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:03.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:03.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:03.680+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:03.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:03.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:03.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:03.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:03.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:03.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:03.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:03.435+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:03.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:03.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:03.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:03.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:03.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:03.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:03.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:03.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:04.665 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:04.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:04.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:04.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:04.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:04.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:04.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:04.388+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:04.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:04.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:04.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:04.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:04.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:04.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:04.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:04.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:04.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:04.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:04.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:05.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:04.664+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:05.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:05.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:05.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:05.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:05.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:05.670+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:05.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:05.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:05.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:05.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:05.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:05.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:05.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:05.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:05.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:05.502+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:05.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:05.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:05.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:05.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:05.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:05.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:05.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:05.433+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:06.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:05.687+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50808 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:06.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:06.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:06.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:06.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:06.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:06.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:06.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:06.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:06.710 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:06.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:06.710 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:06.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:06.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:06.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:06.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:06.436+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:06.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:06.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:06.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:06.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:07.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:07.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:07.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:07.531+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:07.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:07.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:07.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:07.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:07.681+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:07.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:07.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:07.455+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:07.456+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:08.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:08.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:08.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:08.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:08.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:08.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:08.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:08.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:08.580+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:08.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:08.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:08.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:08.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:08.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:08.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:08.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:08.473+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:09.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:09.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:09.748 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:09.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:09.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:09.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:09.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:09.454+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:09.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:09.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:09.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:09.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:09.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:09.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:09.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:09.612+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:10.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:09.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:10.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:09.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:10.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:09.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:10.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:09.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:10.734 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:10.693+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50847 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:10.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:10.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:10.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:10.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:10.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:10.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:10.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:10.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:10.735 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:10.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:10.735 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:10.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:10.735 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:10.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:10.735 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:10.478+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:11.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:10.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:11.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:10.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:10.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:10.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:11.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:11.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:11.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:11.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:11.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:11.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:11.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:11.430+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:11.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:11.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:11.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:11.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:11.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:11.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:11.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:11.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:11.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:11.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:11.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:12.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:11.745+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:12.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:12.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:12.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:12.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:12.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:12.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:12.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:12.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:12.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:12.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:12.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:12.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:12.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:12.449+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:12.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:12.450+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:12.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:12.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:12.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:12.757+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:13.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:13.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:13.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:13.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:13.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:13.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:13.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:13.538+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:13.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:13.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:13.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:13.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:13.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:13.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:13.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:13.413+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:13.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:13.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:13.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:13.753+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:14.756 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:14.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:14.756 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:14.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:14.757 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:14.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:14.757 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:14.461+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:14.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:14.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:14.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:14.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:14.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:14.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:14.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:14.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:14.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:14.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:14.753+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:14.754+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:15.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:15.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:15.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:15.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:15.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:15.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:15.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:15.431+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:15.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:15.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:15.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:15.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:15.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:15.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:15.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:15.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:16.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:15.699+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50886 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:15.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:15.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:15.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:15.713+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:16.744 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:16.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:16.744 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:16.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:16.744 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:16.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:16.744 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:16.459+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:16.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:16.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:16.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:16.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:16.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:16.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:16.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:16.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:16.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:16.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:16.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:16.743+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:17.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:17.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:17.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:17.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:17.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:17.717+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:17.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:17.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:17.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:17.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:17.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:17.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:17.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:17.468+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:17.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:17.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:17.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:17.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:17.630+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:18.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:18.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:18.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:18.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:18.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:18.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:18.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:18.427+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:18.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:18.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:18.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:18.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:18.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:18.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:18.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:18.639+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:18.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:18.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:18.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:18.691+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:19.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:19.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:19.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:19.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:19.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:19.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:19.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:19.657+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:19.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:19.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:19.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:19.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:19.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:19.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:19.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:19.449+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:20.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:19.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:20.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:19.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:19.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:19.722+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:20.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:20.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:20.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:20.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:20.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:20.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:20.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:20.420+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:20.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:20.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:20.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:20.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:20.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:20.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:20.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:20.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:21.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:20.706+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50925 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:20.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:20.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:20.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:20.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:21.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:21.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:21.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:21.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:21.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:21.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:21.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:21.411+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:21.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:21.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:21.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:21.665+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:21.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:21.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:21.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:22.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:21.689+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:22.653 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:22.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:22.653 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:22.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:22.653 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:22.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:22.653 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:22.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:22.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:22.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:22.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:22.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:22.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:22.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:22.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:22.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:23.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:23.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:23.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:23.380+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:23.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:23.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:23.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:23.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:23.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:23.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:23.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:23.639+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:24.654 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:24.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:24.654 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:24.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:24.654 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:24.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:24.654 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:24.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:24.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:24.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:24.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:24.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:24.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:24.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:24.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:24.627+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:24.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:24.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:24.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:24.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:25.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:25.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:25.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:25.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:25.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:25.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:25.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:25.362+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:25.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:25.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:25.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:25.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:25.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:25.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:25.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:25.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:26.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:25.712+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 50964 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:25.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:25.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:25.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:25.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:26.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:26.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:26.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:26.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:26.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:26.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:26.651 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:26.327+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:26.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:27.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:26.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:27.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:26.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:27.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:26.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:26.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:26.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:26.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:26.649+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:27.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:27.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:27.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:27.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:27.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:27.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:27.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:27.355+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:27.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:27.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:27.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:27.666+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:27.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:27.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:27.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:27.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:28.660 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:28.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:28.660 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:28.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:28.660 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:28.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:28.660 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:28.309+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:28.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:28.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:28.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:28.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:28.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:28.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:28.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:28.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:29.705 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:29.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:29.705 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:29.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:29.705 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:29.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:29.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:29.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:29.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:30.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:29.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:30.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:29.703+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:30.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:29.704+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:29.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:29.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:29.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:30.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:29.717+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:30.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:30.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:30.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:30.348+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:31.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:30.718+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51003 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:31.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:30.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:31.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:30.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:31.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:30.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:31.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:30.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:31.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:30.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:31.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:30.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:31.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:30.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:31.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:30.685+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:31.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:31.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:31.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:31.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:31.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:31.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:31.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:31.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:32.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:31.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:31.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:31.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:31.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:31.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:31.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:32.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:31.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:32.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:31.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:32.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:32.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:32.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:32.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:32.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:32.298+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:32.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:32.298+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:33.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:32.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:32.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:32.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:32.644+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:33.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:32.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:33.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:32.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:33.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:32.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:33.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:32.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:33.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:33.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:33.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:33.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:33.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:33.312+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:33.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:33.313+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:33.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:33.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:33.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:33.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:33.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:33.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:33.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:33.667+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:34.644 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:34.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:34.644 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:34.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:34.644 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:34.355+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:34.644 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:34.356+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:34.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:34.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:34.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:34.716+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:34.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:35.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:34.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:35.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:34.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:35.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:34.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:35.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:35.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:35.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:35.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:35.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:35.326+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:35.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:35.327+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:36.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:35.725+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51042 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:35.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:35.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:35.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:35.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:35.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:35.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:35.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:36.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:35.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:36.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:36.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:36.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:36.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:36.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:36.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:36.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:36.352+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:36.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:36.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:36.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:36.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:36.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:36.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:36.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:37.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:36.656+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:37.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:37.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:37.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:37.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:37.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:37.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:37.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:37.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:37.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:37.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:37.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:37.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:37.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:37.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:37.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:37.361+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:37.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:37.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:37.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:37.700+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:38.651 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:38.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:38.651 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:38.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:38.651 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:38.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:38.651 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:38.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:38.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:38.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:38.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:38.674+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:38.649+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:38.649+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:39.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:38.649+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:39.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:38.649+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:39.721 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:39.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:39.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:39.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:39.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:39.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:39.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:39.635+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:39.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:39.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:39.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:39.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:39.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:39.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:39.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:39.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:40.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:39.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:39.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:39.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:39.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:40.733 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:40.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:40.733 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:40.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:40.733 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:40.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:40.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:40.403+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:40.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:40.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:40.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:40.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:40.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:40.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:40.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:40.623+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:41.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:40.731+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51081 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:41.077 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:40.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:41.078 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:40.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:41.078 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:40.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:41.078 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:40.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:41.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:41.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:41.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:41.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:41.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:41.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:41.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:41.598+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:41.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:41.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:41.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:41.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:41.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:41.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:41.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:41.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:41.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:41.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:41.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:41.770+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:42.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:42.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:42.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:42.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:42.605+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:42.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:42.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:42.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:42.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:42.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:42.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:42.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:42.461+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:42.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:42.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:42.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:42.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:43.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:43.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:43.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:43.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:43.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:43.603+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:43.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:43.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:43.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:43.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:43.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:43.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:43.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:43.491+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:43.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:43.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:43.841+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:43.842+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:44.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:44.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:44.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:44.617+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:44.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:44.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:44.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:44.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:44.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:44.493+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:44.795 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:44.493+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:44.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:44.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:44.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:44.819+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:45.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:45.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:45.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:45.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:45.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:45.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:45.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:45.587+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:45.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:45.462+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:45.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:45.462+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:45.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:45.462+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:45.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:45.462+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:46.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:45.737+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51120 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:45.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:45.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:45.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:45.860+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:46.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:46.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:46.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:46.442+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:46.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:46.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:46.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:46.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:46.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:46.550+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:46.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:46.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:47.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:46.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:47.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:46.889+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:47.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:47.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:47.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:47.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:47.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:47.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:47.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:47.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:47.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:47.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:47.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:47.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:47.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:47.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:47.916+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:47.917+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:48.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:48.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:48.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:48.433+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:48.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:48.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:48.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:48.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:49.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:48.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:48.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:48.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:48.946+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:49.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:49.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:49.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:49.616+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:49.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:49.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:49.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:49.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:49.409+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:49.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:49.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:49.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:49.942+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:50.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:50.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:50.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:50.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:50.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:50.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:50.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:50.582+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:50.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:50.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:50.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:50.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:50.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:50.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:50.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:50.444+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:51.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:50.744+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51159 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:50.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:50.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:50.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:50.927+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:51.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:51.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:51.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:51.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:51.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:51.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:51.441+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:51.442+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:51.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:51.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:51.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:51.919+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:52.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:52.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:52.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:52.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:52.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:52.404+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:52.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:52.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:52.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:52.529+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:52.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:52.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:52.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:52.952+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:53.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:53.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:53.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:53.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:53.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:53.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:53.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:53.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:53.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:53.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:53.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:53.421+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:53.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:53.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:53.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:53.983+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:54.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:54.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:54.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:54.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:54.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:54.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:54.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:54.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:54.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:54.513+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:55.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:55.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:55.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:55.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:55.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:55.022+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:55.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:55.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:55.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:55.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:55.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:55.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:55.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:55.464+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:55.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:55.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:55.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:55.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:55.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:55.361+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:55.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:55.362+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:56.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:00:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:00:55.750+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51198 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:00:56.424 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:56.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:56.424 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:56.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:56.424 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:56.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:56.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:56.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:56.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:56.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:56.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:56.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:56.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:56.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:56.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:56.319+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:56.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:56.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:56.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:56.421+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:57.375 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:57.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:57.376 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:57.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:57.376 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:57.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:57.376 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:57.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:57.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:57.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:57.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:57.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:57.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:57.289+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:57.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:57.289+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:57.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:57.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:57.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:57.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:58.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:58.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:58.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:58.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:58.332+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:58.332+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:58.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:58.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:58.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:58.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:58.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:58.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:58.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:58.254+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:58.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:58.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:58.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:58.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:58.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:58.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:58.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:58.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:59.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:59.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:59.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:59.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:59.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:59.108+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:00:59.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:00:59.109+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:00:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:59.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:59.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:59.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:00:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:00:59.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:00:59.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:59.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:59.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:59.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:59.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:59.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:00:59.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:00:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:00:59.344+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:00.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:00.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:00.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:00.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:00.287+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:00.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:00.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:00.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:00.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:00.306+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:00.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:00.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:00.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:00.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:00.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:00.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:00.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:00.121+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:01.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:00.756+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51237 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:01.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:01.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:01.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:01.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:01.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:01.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:01.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:01.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:01.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:01.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:01.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:01.145+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:01.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:01.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:01.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:01.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:01.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:01.264+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:01.545 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:01.265+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:02.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:02.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:02.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:02.146+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:02.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:02.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:02.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:02.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:02.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:02.281+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:02.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:02.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:02.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:02.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:02.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:02.246+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:03.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:03.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:03.200+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:03.201+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:03.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:03.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:03.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:03.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:03.130+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:03.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:03.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:03.325+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:03.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:03.325+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:04.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:04.114+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:04.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:04.114+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:04.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:04.114+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:04.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:04.115+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:04.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:04.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:04.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:04.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:04.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:04.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:04.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:04.192+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:04.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:04.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:04.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:04.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:04.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:04.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:04.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:04.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:05.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:05.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:05.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:05.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:05.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:05.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:05.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:05.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:05.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:05.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:05.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:05.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:05.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:05.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:05.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:05.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:05.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:05.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:05.219+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:05.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:05.219+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:06.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:05.762+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51276 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:06.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:06.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:06.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:06.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:06.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:06.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:06.047+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:06.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:06.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:06.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:06.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:06.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:06.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:06.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:06.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:06.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:06.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:06.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:06.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:06.289+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:07.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:07.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:07.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:07.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:07.081+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:07.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:07.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:07.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:07.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:07.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:07.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:07.222+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:07.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:07.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:07.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:07.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:07.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:07.283+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:07.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:07.284+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:08.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:08.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:08.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:08.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:08.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:08.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:08.444 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:08.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:08.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:08.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:08.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:08.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:08.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:08.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:08.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:08.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:08.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:08.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:08.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:08.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:08.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:08.070+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:08.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:08.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:09.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:09.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:09.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:09.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:09.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:09.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:09.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:09.204+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:09.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:09.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:09.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:09.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:09.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:09.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:09.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:09.207+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:09.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:09.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:09.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:09.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:09.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:09.060+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:10.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:10.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:10.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:10.220+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:10.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:10.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:10.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:10.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:10.094+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:10.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:10.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:10.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:10.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:10.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:10.171+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:10.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:10.172+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:11.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:10.769+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51315 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:11.074+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:11.074+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:11.074+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:11.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:11.075+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:11.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:11.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:11.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:11.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:11.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:11.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:11.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:11.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:11.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:11.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:11.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:11.170+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:11.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:11.171+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:12.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:12.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:12.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:12.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:12.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:12.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:12.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:12.121+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:12.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:12.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:12.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:12.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:12.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:12.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:12.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:12.253+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:12.403 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:12.183+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:12.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:12.183+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:12.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:12.183+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:12.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:12.184+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:13.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:13.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:13.093+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:13.093+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:13.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:13.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:13.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:13.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:13.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:13.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:13.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:13.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:13.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:13.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:13.156+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:13.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:13.156+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:14.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:14.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:14.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:14.273+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:14.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:14.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:14.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:14.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:14.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:14.060+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:14.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:14.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:14.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:14.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:14.118+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:15.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:15.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:15.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:15.105+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:15.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:15.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:15.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:15.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:15.072+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:15.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:15.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:15.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:15.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:15.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:15.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:15.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:15.319+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:16.041 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:15.775+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51354 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:16.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:16.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:16.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:16.041+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:16.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:16.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:16.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:16.081+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:16.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:16.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:16.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:16.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:16.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:16.328+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:16.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:16.329+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:17.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:17.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:17.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:17.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:17.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:17.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:17.066+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:17.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:17.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:17.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:17.036+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:17.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:17.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:17.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:17.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:17.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:17.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:17.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:17.335+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:18.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:18.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:18.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:18.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:18.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:18.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:18.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:18.059+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:18.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:17.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:18.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:17.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:18.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:17.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:18.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:17.996+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:18.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:18.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:18.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:18.287+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:19.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:19.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:19.100+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:19.101+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:18.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:18.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:19.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:18.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:19.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:18.997+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:19.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:19.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:19.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:19.304+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:20.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:20.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:20.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:20.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:20.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:20.127+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:20.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:20.128+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:20.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:19.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:20.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:19.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:20.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:19.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:20.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:19.964+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:20.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:20.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:20.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:20.266+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:21.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:20.782+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51393 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:21.008+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:21.008+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:21.008+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:21.009+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:21.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:21.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:21.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:21.288+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:21.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:21.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:21.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:21.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:21.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:21.105+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:22.260 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:22.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:22.260 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:22.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:22.260 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:22.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:22.260 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:22.066+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:22.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:21.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:22.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:21.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:22.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:21.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:22.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:21.962+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:22.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:22.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:22.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:22.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:23.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:23.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:23.258 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:23.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:23.258 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:23.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:23.258 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:23.107+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:23.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:22.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:23.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:22.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:23.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:22.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:23.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:22.944+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:23.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:23.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:23.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:23.256+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:24.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:24.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:24.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:24.067+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:23.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:23.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:23.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:24.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:23.991+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:24.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:24.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:24.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:24.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:24.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:24.296+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:24.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:24.297+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:25.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:25.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:25.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:25.027+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:25.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:25.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:25.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:25.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:25.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:25.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:25.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:25.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:25.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:25.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:25.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:25.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:25.052+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:26.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:25.789+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51432 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:26.007+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:26.007+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:26.007+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:26.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:26.007+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:26.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:26.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:26.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:26.086+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:26.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:26.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:26.293+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:26.293+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:26.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:26.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:26.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:26.971+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:27.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:27.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:27.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:27.102+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:27.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:27.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:27.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:27.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:27.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:27.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:27.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:27.330+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:28.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:27.977+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:27.977+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:27.977+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:27.977+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:28.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:28.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:28.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:28.085+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:28.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:28.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:28.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:28.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:28.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:28.356+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:29.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:29.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:29.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:29.078+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:29.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:28.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:29.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:28.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:29.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:28.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:29.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:28.991+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:29.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:29.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:29.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:29.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:29.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:29.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:29.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:29.988+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:30.042+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:30.042+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:30.042+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:30.043+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:30.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:30.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:30.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:30.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:30.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:30.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:30.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:30.422+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:31.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:30.795+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51470 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:30.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:30.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:30.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:30.953+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:31.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:30.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:31.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:30.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:31.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:30.999+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:31.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:31.000+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:31.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:31.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:31.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:31.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:31.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:31.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:31.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:31.452+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:31.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:31.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:31.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:31.964+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:31.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:31.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:31.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:31.962+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:32.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:32.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:32.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:32.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:32.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:32.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:32.408+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:32.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:32.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:33.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:32.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:33.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:32.918+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:33.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:33.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:33.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:33.010+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:33.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:33.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:33.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:33.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:33.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:33.389+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:34.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:34.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:34.017+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:34.018+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:33.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:33.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:33.910+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:34.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:33.911+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:34.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:34.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:34.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:34.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:34.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:34.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:34.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:35.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:35.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:35.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:35.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:35.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:35.067+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:34.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:34.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:34.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:34.915+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:35.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:35.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:35.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:35.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:35.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:35.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:36.092 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:35.802+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51510 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:36.092 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:35.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:36.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:35.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:36.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:35.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:36.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:35.913+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:36.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:36.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:36.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:36.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:36.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:36.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:36.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:36.091+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:36.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:36.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:36.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:36.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:36.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:36.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:36.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:36.406+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:37.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:37.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:37.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:37.129+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:37.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:37.129+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:36.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:36.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:36.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:36.929+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:37.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:37.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:37.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:37.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:37.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:37.388+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:37.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:37.389+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:38.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:38.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:38.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:38.085+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:37.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:37.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:37.885+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:37.886+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:38.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:38.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:38.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:38.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:38.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:38.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:38.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:38.383+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:38.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:38.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:39.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:38.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:39.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:38.917+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:39.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:39.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:39.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:39.134+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:39.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:39.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:39.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:39.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:39.427+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:40.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:39.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:40.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:39.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:40.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:39.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:40.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:39.897+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:40.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:40.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:40.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:40.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:40.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:40.101+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:40.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:40.102+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:40.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:40.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:40.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:40.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:40.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:40.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:40.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:40.407+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:41.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:40.808+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51548 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:41.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:40.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:41.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:40.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:41.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:40.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:41.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:40.942+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:41.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:41.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:41.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:41.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:41.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:41.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:41.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:41.133+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:41.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:41.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:41.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:41.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:41.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:41.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:41.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:41.400+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:41.973+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:41.973+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:41.973+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:41.974+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:42.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:42.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:42.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:42.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:42.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:42.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:42.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:42.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:42.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:42.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:42.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:42.410+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:42.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:42.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:42.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:42.958+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:43.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:43.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:43.128+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:43.128+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:43.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:43.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:43.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:43.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:43.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:43.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:43.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:43.401+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:43.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:44.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:43.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:44.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:43.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:44.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:43.930+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:44.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:44.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:44.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:44.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:44.167+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:44.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:44.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:44.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:44.375+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:44.376+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:45.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:44.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:45.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:44.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:45.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:44.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:45.185 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:44.892+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:45.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:45.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:45.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:45.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:45.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:45.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:45.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:45.184+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:45.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:45.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:45.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:45.370+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:45.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:45.371+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:46.187 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:45.814+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51588 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:46.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:45.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:46.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:45.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:46.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:45.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:46.188 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:45.854+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:46.185+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:46.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:46.185+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:46.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:46.185+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:46.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:46.186+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:46.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:46.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:46.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:46.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:46.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:46.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:46.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:46.369+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:47.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:46.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:47.162 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:46.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:47.162 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:46.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:47.162 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:46.823+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:47.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:47.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:47.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:47.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:47.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:47.160+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:47.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:47.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:47.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:47.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:47.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:47.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:47.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:47.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:47.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:47.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:47.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:47.788+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:48.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:48.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:48.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:48.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:48.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:48.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:48.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:48.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:48.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:48.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:49.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:48.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:49.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:48.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:49.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:48.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:49.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:48.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:49.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:49.364+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:49.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:49.364+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:49.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:49.364+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:49.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:49.365+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:49.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:49.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:49.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:49.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:49.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:49.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:49.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:49.200+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:50.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:49.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:50.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:49.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:50.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:49.811+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:50.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:49.812+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:50.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:50.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:50.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:50.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:50.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:50.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:50.163+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:50.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:50.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:50.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:50.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:50.382+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:51.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:50.820+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51627 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:51.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:50.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:50.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:50.785+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:51.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:50.786+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:51.392 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:51.134+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:51.392 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:51.134+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:51.392 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:51.134+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:51.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:51.134+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:51.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:51.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:51.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:51.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:51.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:51.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:51.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:51.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:52.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:51.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:51.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:51.745+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:52.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:51.745+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:52.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:52.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:52.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:52.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:52.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:52.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:52.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:52.111+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:52.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:52.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:52.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:52.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:52.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:52.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:52.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:52.400+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:53.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:52.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:53.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:52.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:52.743+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:52.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:53.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:53.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:53.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:53.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:53.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:53.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:53.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:53.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:53.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:53.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:53.365+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:53.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:53.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:53.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:53.781+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:54.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:54.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:54.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:54.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:54.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:54.402+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:54.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:54.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:54.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:54.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:54.187+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:55.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:54.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:55.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:54.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:55.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:54.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:55.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:54.806+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:55.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:55.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:55.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:55.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:55.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:55.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:55.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:55.216+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:55.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:55.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:55.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:55.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:55.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:55.448+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:55.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:55.449+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:56.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:01:55.827+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51666 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:01:56.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:55.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:56.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:55.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:56.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:55.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:56.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:55.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:56.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:56.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:56.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:56.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:56.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:56.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:56.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:56.197+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:56.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:56.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:56.731+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:56.731+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:56.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:56.494+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:56.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:56.494+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:56.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:56.494+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:56.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:56.495+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:57.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:57.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:57.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:57.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:57.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:57.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:57.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:57.217+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:57.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:57.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:57.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:57.718+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:57.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:57.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:57.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:57.507+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:58.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:58.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:58.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:58.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:58.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:58.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:58.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:58.191+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:58.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:58.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:58.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:58.741+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:58.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:58.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:58.495+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:58.496+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:59.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:59.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:59.494 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:59.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:59.494 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:59.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:01:59.494 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:01:59.224+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:01:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:59.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:59.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:59.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:01:59.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:01:59.492+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:01:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:59.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:59.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:59.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:01:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:01:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:01:59.710+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:00.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:00.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:00.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:00.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:00.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:00.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:00.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:00.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:00.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:00.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:00.491+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:00.492+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:00.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:00.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:00.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:00.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:00.708+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:01.204 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:00.833+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51705 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:01.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:01.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:01.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:01.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:01.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:01.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:01.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:01.203+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:01.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:01.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:01.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:01.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:01.667+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:01.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:01.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:01.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:01.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:01.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:01.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:01.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:01.506+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:02.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:02.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:02.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:02.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:02.235+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:02.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:02.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:02.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:02.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:02.484+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:02.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:02.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:02.656+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:03.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:02.657+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:03.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:03.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:03.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:03.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:03.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:03.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:03.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:03.201+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:03.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:03.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:03.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:03.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:03.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:03.503+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:03.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:03.504+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:03.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:03.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:03.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:03.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:04.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:04.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:04.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:04.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:04.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:04.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:04.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:04.153+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:04.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:04.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:04.664+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:04.665+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:04.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:04.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:04.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:04.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:04.472+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:05.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:05.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:05.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:05.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:05.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:05.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:05.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:05.144+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:05.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:05.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:05.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:05.516+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:05.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:05.517+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:05.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:05.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:05.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:05.680+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:06.165 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:05.840+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51744 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:06.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:06.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:06.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:06.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:06.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:06.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:06.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:06.164+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:06.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:06.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:06.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:06.635+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:06.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:06.534+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:06.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:06.534+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:06.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:06.534+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:06.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:06.534+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:07.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:07.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:07.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:07.142+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:07.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:07.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:07.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:07.625+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:07.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:07.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:07.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:07.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:07.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:07.526+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:08.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:08.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:08.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:08.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:08.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:08.104+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:08.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:08.105+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:08.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:08.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:08.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:08.601+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:08.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:08.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:08.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:08.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:08.526+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:09.138+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:09.138+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:09.138+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:09.139+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:09.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:09.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:09.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:09.601+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:09.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:09.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:09.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:09.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:09.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:09.554+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:10.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:10.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:10.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:10.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:10.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:10.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:10.865 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:10.846+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51783 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:10.866 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:10.632+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:10.866 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:10.632+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:10.866 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:10.632+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:10.866 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:10.632+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:10.866 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:10.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:10.866 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:10.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:10.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:10.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:10.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:10.603+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:11.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:11.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:11.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:11.059+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:11.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:11.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:11.678+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:12.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:11.679+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:11.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:11.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:12.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:11.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:12.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:11.597+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:12.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:12.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:12.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:12.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:12.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:13.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:12.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:12.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:12.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:13.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:12.637+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:12.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:12.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:12.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:13.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:12.591+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:13.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:13.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:13.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:13.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:13.089+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:14.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:13.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:13.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:13.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:14.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:13.593+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:14.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:13.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:14.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:13.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:14.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:13.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:14.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:13.572+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:14.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:14.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:14.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:14.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:14.112+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:14.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:14.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:14.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:14.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:14.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:14.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:14.608+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:15.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:14.609+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:15.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:15.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:15.124+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:15.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:15.125+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:15.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:15.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:15.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:15.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:15.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:15.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:15.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:15.587+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:15.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:15.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:15.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:15.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:15.856 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:15.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:15.856 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:15.630+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:16.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:15.852+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51822 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:16.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:16.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:16.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:16.098+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:16.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:16.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:16.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:16.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:16.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:16.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:16.609+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:16.609+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:17.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:17.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:17.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:17.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:17.052+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:18.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:17.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:18.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:17.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:18.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:17.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:18.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:17.635+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:18.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:17.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:18.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:17.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:18.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:17.614+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:18.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:17.615+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:18.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:18.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:18.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:18.008+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:19.008 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:18.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:19.008 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:18.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:19.008 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:18.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:19.008 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:18.631+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:19.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:18.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:19.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:18.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:19.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:18.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:19.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:18.644+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:19.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:19.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:19.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:19.007+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:19.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:19.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:19.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:19.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:19.686+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:19.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:19.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:19.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:19.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:20.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:19.967+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:20.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:19.967+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:20.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:19.967+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:20.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:19.968+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:20.979 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:20.859+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51861 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:20.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:20.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:20.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:20.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:20.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:20.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:20.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:20.725+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:20.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:20.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:20.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:20.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:20.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:20.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:20.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:20.636+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:21.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:20.988+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:21.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:20.988+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:21.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:20.988+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:21.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:20.989+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:21.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:21.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:21.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:21.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:21.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:21.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:21.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:21.727+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:21.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:21.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:21.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:21.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:21.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:21.598+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:21.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:21.599+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:21.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:21.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:21.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:22.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:21.984+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:22.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:22.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:22.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:22.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:22.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:22.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:22.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:22.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:22.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:22.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:22.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:22.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:22.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:22.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:22.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:22.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:22.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:22.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:23.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:22.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:23.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:22.938+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:23.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:23.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:23.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:23.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:23.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:23.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:23.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:23.549+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:23.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:23.718+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:23.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:23.718+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:23.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:23.718+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:23.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:23.719+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:23.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:23.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:23.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:23.920+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:24.876 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:24.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:24.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:24.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:24.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:24.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:24.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:24.587+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:24.877 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:24.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:24.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:24.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:24.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:24.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:24.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:24.759+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:24.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:24.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:24.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:24.875+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:25.867 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:25.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:25.867 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:25.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:25.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:25.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:25.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:25.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:25.868 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:25.742+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:25.868 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:25.742+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:25.868 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:25.742+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:25.868 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:25.742+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:26.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:25.865+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51900 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:26.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:25.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:26.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:25.922+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:26.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:25.922+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:26.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:25.923+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:26.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:26.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:26.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:26.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:26.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:26.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:26.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:26.605+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:26.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:26.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:26.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:26.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:26.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:26.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:26.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:26.726+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:26.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:26.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:26.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:26.921+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:27.968 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:27.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:27.968 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:27.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:27.969 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:27.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:27.969 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:27.740+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:27.969 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:27.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:27.969 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:27.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:27.969 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:27.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:27.969 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:27.611+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:27.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:27.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:27.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:27.967+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:28.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:28.593+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:28.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:28.593+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:28.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:28.593+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:28.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:28.594+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:28.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:28.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:28.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:28.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:28.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:28.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:28.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:28.700+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:28.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:28.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:28.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:28.952+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:29.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:29.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:29.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:29.570+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:29.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:29.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:29.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:29.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:29.714+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:29.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:29.715+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:30.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:29.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:30.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:29.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:29.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:30.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:29.980+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:30.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:30.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:30.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:30.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:30.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:30.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:30.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:30.718+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:30.875 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:30.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:30.875 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:30.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:30.875 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:30.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:30.875 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:30.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:31.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:30.872+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51939 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:30.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:30.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:30.950+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:30.951+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:31.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:31.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:31.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:31.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:31.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:31.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:31.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:31.722+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:31.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:31.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:31.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:31.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:31.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:31.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:31.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:31.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:32.274 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:31.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:32.274 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:31.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:32.274 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:31.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:32.274 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:31.987+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:32.958 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:32.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:32.959 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:32.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:32.959 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:32.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:32.959 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:32.544+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:32.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:32.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:32.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:32.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:32.959 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:32.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:32.960 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:32.707+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:32.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:32.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:32.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:32.958+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:33.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:33.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:33.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:33.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:33.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:33.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:33.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:33.680+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:33.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:33.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:33.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:33.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:33.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:33.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:33.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:33.513+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:33.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:33.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:33.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:33.959+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:34.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:34.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:34.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:34.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:34.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:34.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:34.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:34.695+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:34.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:34.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:34.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:34.538+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:34.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:34.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:35.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:34.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:35.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:34.984+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:35.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:35.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:35.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:35.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:35.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:35.646+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:35.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:35.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:35.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:35.536+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:36.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:35.878+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 51978 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:36.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:36.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:36.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:36.021+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:36.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:36.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:36.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:36.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:36.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:36.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:36.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:36.622+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:36.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:36.544+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:36.998 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:36.544+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:36.998 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:36.544+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:36.998 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:36.545+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:36.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:36.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:36.995+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:37.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:36.996+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:37.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:37.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:37.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:37.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:37.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:37.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:37.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:37.649+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:37.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:37.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:37.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:37.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:37.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:37.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:37.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:37.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:37.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:37.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:37.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:37.950+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:38.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:38.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:38.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:38.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:38.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:38.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:38.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:38.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:38.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:38.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:38.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:38.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:38.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:38.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:38.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:38.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:38.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:38.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:38.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:38.976+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:39.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:39.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:39.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:39.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:39.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:39.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:39.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:39.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:39.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:39.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:39.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:39.688+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:40.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:39.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:40.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:39.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:39.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:39.965+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:40.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:40.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:40.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:40.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:40.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:40.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:40.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:40.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:40.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:40.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:40.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:40.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:40.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:40.646+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:40.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:40.646+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:41.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:40.885+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52017 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:41.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:40.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:41.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:40.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:41.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:40.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:41.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:40.917+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:41.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:41.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:41.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:41.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:41.540+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:41.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:41.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:41.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:41.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:41.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:41.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:41.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:41.618+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:42.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:41.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:42.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:41.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:41.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:41.946+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:42.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:42.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:42.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:42.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:42.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:42.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:42.990 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:42.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:42.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:42.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:42.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:42.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:42.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:42.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:42.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:42.631+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:42.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:42.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:42.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:42.988+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:43.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:43.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:43.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:43.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:43.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:43.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:43.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:43.626+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:43.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:43.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:43.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:43.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:43.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:43.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:43.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:43.599+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:43.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:43.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:43.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:43.983+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:45.011 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:44.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:45.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:44.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:45.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:44.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:45.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:44.593+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:45.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:44.560+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:45.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:44.560+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:45.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:44.560+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:45.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:44.561+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:45.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:45.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:45.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:45.010+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:45.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:45.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:45.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:45.576+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:45.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:45.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:45.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:45.524+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:46.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:45.891+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52057 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:46.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:46.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:46.016+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:46.017+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:46.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:46.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:46.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:46.541+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:46.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:46.542+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:46.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:46.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:46.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:46.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:46.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:47.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:46.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:47.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:46.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:47.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:46.971+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:47.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:47.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:47.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:47.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:47.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:47.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:47.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:47.519+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:47.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:47.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:47.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:47.485+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:47.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:47.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:47.955+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:47.956+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:48.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:48.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:48.502+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:48.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:48.502+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:48.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:48.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:48.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:48.467+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:49.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:48.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:48.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:48.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:48.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:49.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:49.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:49.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:49.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:49.476+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:49.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:49.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:49.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:49.418+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:49.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:49.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:49.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:50.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:49.952+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:50.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:50.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:50.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:50.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:50.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:50.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:50.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:50.388+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:50.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:50.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:50.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:50.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:50.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:50.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:50.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:50.430+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:51.018 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:50.898+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52096 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:51.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:50.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:51.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:50.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:51.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:50.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:51.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:50.908+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:51.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:51.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:51.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:51.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:51.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:51.395+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:51.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:51.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:51.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:51.398+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:51.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:51.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:51.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:51.866+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:52.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:52.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:52.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:52.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:52.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:52.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:52.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:52.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:52.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:52.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:52.435+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:52.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:52.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:52.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:53.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:52.859+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:53.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:53.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:53.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:53.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:53.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:53.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:53.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:53.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:53.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:53.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:53.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:53.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:53.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:53.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:53.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:54.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:54.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:54.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:54.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:54.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:54.442+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:54.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:54.442+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:54.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:54.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:54.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:54.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:54.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:54.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:54.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:54.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:55.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:54.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:55.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:54.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:54.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:54.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:55.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:55.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:55.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:55.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:55.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:55.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:55.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:55.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:55.757 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:55.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:55.757 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:55.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:55.757 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:55.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:55.757 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:55.369+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:56.025 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:02:55.905+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52135 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:02:56.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:55.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:56.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:55.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:56.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:55.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:56.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:55.754+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:56.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:56.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:56.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:56.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:56.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:56.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:56.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:56.446+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:56.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:56.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:56.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:56.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:56.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:56.405+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:57.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:56.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:57.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:56.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:56.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:56.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:57.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:57.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:57.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:57.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:57.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:57.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:57.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:57.478+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:57.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:57.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:57.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:57.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:57.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:57.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:57.740 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:57.446+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:58.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:57.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:58.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:57.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:58.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:57.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:58.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:57.737+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:58.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:58.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:58.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:58.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:58.732 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:58.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:58.732 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:58.516+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:58.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:58.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:58.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:58.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:58.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:58.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:58.732 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:58.397+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:58.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:58.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:59.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:58.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:02:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:58.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:02:59.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:59.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:59.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:59.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:59.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:59.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:02:59.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:02:59.397+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:02:59.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:59.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:59.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:59.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:59.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:59.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:02:59.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:02:59.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:00.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:59.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:59.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:59.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:02:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:02:59.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:00.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:00.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:00.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:00.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:00.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:00.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:00.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:00.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:00.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:00.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:00.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:00.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:00.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:00.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:00.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:00.721+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:01.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:00.911+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52174 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:01.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:01.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:01.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:01.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:01.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:01.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:01.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:01.539+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:01.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:01.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:01.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:01.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:01.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:01.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:01.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:01.338+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:02.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:01.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:02.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:01.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:01.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:01.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:02.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:02.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:02.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:02.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:02.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:02.492+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:02.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:02.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:02.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:02.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:02.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:02.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:02.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:02.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:02.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:02.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:02.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:02.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:02.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:02.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:02.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:02.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:02.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:02.679+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:03.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:03.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:03.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:03.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:03.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:03.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:03.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:03.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:03.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:03.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:03.363+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:03.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:03.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:03.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:04.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:03.726+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:04.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:04.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:04.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:04.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:04.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:04.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:04.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:04.334+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:04.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:04.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:04.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:04.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:04.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:04.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:04.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:04.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:04.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:04.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:04.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:04.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:05.592 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:05.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:05.592 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:05.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:05.592 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:05.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:05.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:05.328+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:05.919 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:05.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:05.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:05.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:05.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:05.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:05.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:05.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:05.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:05.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:05.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:05.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:05.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:05.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:05.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:05.591+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:06.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:05.918+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52213 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:06.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:06.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:06.743 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:06.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:06.743 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:06.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:06.743 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:06.576+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:06.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:06.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:06.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:06.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:06.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:06.338+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:06.744 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:06.339+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:06.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:06.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:06.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:06.741+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:07.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:07.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:07.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:07.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:07.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:07.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:07.628 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:07.354+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:07.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:07.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:07.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:07.625+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:07.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:07.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:07.752+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:07.753+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:08.600 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:08.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:08.601 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:08.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:08.601 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:08.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:08.601 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:08.308+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:08.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:09.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:08.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:09.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:08.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:09.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:08.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:09.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:08.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:09.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:08.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:09.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:08.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:09.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:08.599+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:09.590 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:09.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:09.590 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:09.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:09.590 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:09.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:09.590 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:09.292+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:10.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:09.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:10.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:09.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:10.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:09.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:10.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:09.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:10.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:09.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:10.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:09.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:10.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:09.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:10.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:09.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:10.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:10.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:10.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:10.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:10.300+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:10.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:10.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:10.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:10.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:10.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:10.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:10.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:10.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:10.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:10.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:10.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:10.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:10.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:10.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:10.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:10.562+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:11.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:10.924+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52252 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:11.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:11.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:11.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:11.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:11.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:11.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:11.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:11.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:11.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:11.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:11.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:11.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:11.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:11.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:11.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:11.350+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:11.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:11.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:11.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:12.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:11.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:12.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:12.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:12.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:12.348+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:12.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:12.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:12.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:12.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:12.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:12.776+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:12.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:12.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:12.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:12.540+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:13.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:13.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:13.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:13.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:13.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:13.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:13.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:13.583+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:13.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:13.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:13.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:13.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:13.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:13.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:13.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:13.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:13.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:13.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:13.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:13.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:14.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:14.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:14.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:14.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:14.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:14.322+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:14.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:14.323+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:14.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:14.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:14.591+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:15.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:14.591+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:14.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:14.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:14.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:14.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:15.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:15.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:15.287+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:15.288+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:15.823 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:15.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:15.824 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:15.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:15.824 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:15.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:15.824 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:15.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:16.269 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:15.930+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52291 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:16.269 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:15.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:16.269 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:15.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:16.269 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:15.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:16.269 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:15.823+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:16.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:16.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:16.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:16.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:16.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:17.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:16.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:16.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:17.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:16.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:16.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:16.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:16.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:16.802+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:17.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:17.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:17.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:17.269+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:18.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:17.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:17.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:17.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:17.630+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:17.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:17.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:17.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:17.846+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:18.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:18.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:18.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:18.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:18.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:18.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:18.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:18.306+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:18.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:18.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:18.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:18.891+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:18.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:18.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:18.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:19.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:18.680+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:19.722 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:19.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:19.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:19.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:19.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:19.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:19.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:19.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:19.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:19.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:19.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:20.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:19.721+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:19.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:19.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:19.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:19.909+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:20.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:20.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:20.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:20.382+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:21.042 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:20.937+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52330 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:20.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:20.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:20.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:21.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:20.764+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:20.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:21.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:20.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:21.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:20.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:21.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:20.941+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:21.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:21.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:21.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:21.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:21.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:21.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:21.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:21.347+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:21.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:21.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:21.740+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:22.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:21.740+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:21.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:21.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:21.912+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:21.913+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:22.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:22.333+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:22.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:22.333+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:22.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:22.333+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:22.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:22.334+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:22.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:22.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:22.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:22.883+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:22.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:22.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:22.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:23.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:22.719+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:23.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:23.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:23.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:23.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:23.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:23.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:23.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:23.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:23.894+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:23.894+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:23.894+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:23.895+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:23.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:23.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:23.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:24.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:23.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:24.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:24.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:24.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:24.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:24.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:24.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:24.665 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:24.328+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:24.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:25.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:24.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:24.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:25.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:24.663+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:24.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:24.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:24.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:24.881+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:25.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:25.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:25.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:25.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:25.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:25.307+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:25.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:25.307+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:25.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:25.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:25.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:25.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:25.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:25.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:25.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:25.657+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:25.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:25.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:25.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:25.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:25.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:25.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:25.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:25.862+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:26.262 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:25.943+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52369 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:26.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:26.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:26.261+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:26.262+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:26.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:26.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:27.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:26.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:27.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:26.854+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:26.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:26.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:26.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:27.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:26.668+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:27.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:27.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:27.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:27.249+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:27.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:27.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:27.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:27.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:27.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:27.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:27.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:28.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:27.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:28.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:28.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:28.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:28.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:28.266+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:29.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:28.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:28.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:28.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:29.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:28.839+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:28.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:28.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:28.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:29.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:28.753+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:29.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:29.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:29.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:29.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:29.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:29.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:29.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:29.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:29.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:30.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:29.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:30.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:29.880+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:30.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:29.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:29.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:29.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:29.758+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:30.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:29.759+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:30.290+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:30.290+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:30.290+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:30.291+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:30.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:30.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:30.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:30.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:30.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:30.724+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:30.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:30.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:30.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:30.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:30.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:30.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:30.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:30.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:30.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:30.841+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:31.246 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:30.950+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52408 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:31.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:31.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:31.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:31.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:31.245+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:31.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:32.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:31.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:31.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:32.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:31.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:31.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:31.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:31.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:32.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:31.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:32.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:32.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:32.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:32.250+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:32.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:32.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:33.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:32.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:33.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:32.769+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:32.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:32.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:32.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:33.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:32.689+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:33.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:33.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:33.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:33.228+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:33.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:33.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:33.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:33.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:33.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:33.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:33.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:34.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:33.658+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:34.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:34.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:34.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:34.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:34.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:34.215+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:34.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:35.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:34.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:34.635+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:35.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:34.636+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:34.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:34.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:34.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:35.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:34.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:35.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:35.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:35.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:35.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:35.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:35.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:35.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:35.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:35.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:35.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:35.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:35.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:35.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:35.959 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:35.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:35.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:35.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:35.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:35.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:35.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:35.612+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:36.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:35.956+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52447 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:36.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:36.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:36.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:36.161+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:36.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:37.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:36.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:36.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:37.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:36.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:36.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:36.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:36.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:36.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:37.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:37.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:37.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:37.187+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:37.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:37.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:37.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:37.688+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:37.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:37.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:37.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:38.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:37.648+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:38.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:38.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:38.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:38.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:38.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:39.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:38.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:38.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:39.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:38.617+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:38.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:38.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:38.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:39.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:38.652+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:39.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:39.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:39.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:39.213+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:39.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:39.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:39.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:39.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:39.667+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:39.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:39.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:39.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:39.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:39.581+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:40.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:40.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:40.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:40.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:40.245+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:40.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:40.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:40.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:40.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:40.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:40.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:40.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:40.675+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:40.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:40.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:40.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:40.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:40.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:40.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:40.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:40.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:41.244 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:40.963+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52486 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:41.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:41.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:41.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:41.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:41.243+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:41.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:41.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:41.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:41.645+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:41.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:41.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:41.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:42.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:41.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:42.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:42.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:42.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:42.219+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:42.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:42.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:42.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:42.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:42.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:42.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:42.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:43.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:42.596+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:43.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:43.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:43.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:43.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:43.236+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:43.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:43.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:43.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:43.580+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:43.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:43.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:43.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:44.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:43.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:44.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:44.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:44.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:44.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:44.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:44.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:44.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:44.218+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:44.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:44.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:44.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:44.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:44.540+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:44.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:44.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:44.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:44.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:45.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:45.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:45.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:45.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:45.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:45.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:45.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:45.248+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:45.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:45.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:45.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:45.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:45.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:45.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:45.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:45.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:45.531+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:45.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:45.532+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:46.261 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:45.969+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52525 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:46.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:46.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:46.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:46.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:46.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:46.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:46.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:46.260+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:46.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:46.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:46.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:46.525+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:46.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:46.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:46.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:46.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:46.528+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:47.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:47.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:47.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:47.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:47.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:47.540+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:47.540+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:47.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:47.540+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:47.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:47.541+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:47.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:47.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:47.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:47.549+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:48.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:48.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:48.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:48.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:48.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:48.263+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:48.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:48.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:48.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:48.560+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:48.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:48.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:48.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:49.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:48.581+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:49.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:49.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:49.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:49.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:49.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:49.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:49.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:49.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:50.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:49.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:49.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:49.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:49.574+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:50.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:49.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:50.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:49.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:50.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:49.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:50.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:49.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:50.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:50.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:50.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:50.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:50.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:50.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:50.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:50.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:50.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:50.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:50.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:50.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:50.609+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:50.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:50.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:50.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:50.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:50.979 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:50.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:50.979 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:50.561+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:51.285 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:50.977+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52564 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:51.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:51.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:51.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:51.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:51.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:51.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:51.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:51.285+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:51.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:51.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:51.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:51.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:51.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:51.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:51.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:51.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:51.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:52.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:52.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:52.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:52.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:52.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:52.252+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:52.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:52.253+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:52.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:53.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:52.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:53.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:52.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:53.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:52.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:52.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:52.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:52.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:53.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:52.571+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:53.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:53.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:53.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:53.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:53.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:53.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:53.230+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:54.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:53.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:54.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:53.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:53.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:53.558+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:53.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:53.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:53.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:54.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:53.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:54.535 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:54.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:54.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:54.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:54.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:54.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:54.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:54.256+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:54.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:54.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:54.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:54.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:54.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:54.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:54.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:54.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:54.580+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:55.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:55.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:55.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:55.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:55.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:55.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:55.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:55.231+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:55.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:55.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:55.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:55.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:55.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:55.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:55.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:55.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:55.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:56.235 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:03:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:03:55.982+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52603 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:03:56.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:56.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:56.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:56.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:56.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:56.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:56.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:56.234+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:56.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:56.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:56.520+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:56.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:56.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:56.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:56.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:56.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:57.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:57.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:57.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:57.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:57.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:57.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:57.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:57.238+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:57.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:57.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:57.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:57.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:57.553+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:57.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:57.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:57.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:57.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:58.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:58.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:58.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:58.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:58.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:58.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:58.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:58.200+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:58.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:58.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:58.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:58.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:58.562+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:58.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:58.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:58.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:58.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:58.498+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:59.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:59.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:59.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:59.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:59.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:59.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:03:59.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:03:59.188+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:03:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:59.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:59.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:59.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:03:59.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:03:59.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:03:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:59.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:59.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:59.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:03:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:03:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:03:59.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:00.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:00.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:00.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:00.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:00.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:00.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:00.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:00.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:00.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:00.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:00.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:00.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:00.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:00.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:00.554+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:00.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:00.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:00.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:00.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:01.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:00.988+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52642 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:01.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:01.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:01.147+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:01.148+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:01.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:01.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:01.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:01.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:01.525+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:01.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:01.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:01.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:01.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:02.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:02.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:02.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:02.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:02.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:02.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:02.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:02.153+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:02.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:02.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:02.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:02.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:02.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:02.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:02.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:02.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:02.563+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:03.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:03.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:03.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:03.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:03.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:03.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:03.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:03.112+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:03.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:03.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:03.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:03.555+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:03.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:03.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:03.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:03.464+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:04.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:04.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:04.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:04.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:04.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:04.117+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:04.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:04.118+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:04.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:04.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:04.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:04.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:04.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:04.580+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:04.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:04.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:04.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:04.425+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:05.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:05.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:05.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:05.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:05.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:05.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:05.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:05.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:05.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:05.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:05.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:05.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:05.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:05.576+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:05.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:05.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:05.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:05.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:05.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:06.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:05.994+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52681 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:06.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:06.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:06.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:06.110+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:06.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:06.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:06.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:06.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:06.582+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:06.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:06.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:06.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:06.479+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:07.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:07.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:07.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:07.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:07.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:07.133+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:07.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:07.134+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:07.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:07.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:07.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:07.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:07.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:07.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:07.533+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:07.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:07.535+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:08.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:08.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:08.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:08.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:08.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:08.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:08.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:08.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:08.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:08.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:08.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:08.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:08.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:08.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:08.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:08.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:08.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:08.440+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:09.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:09.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:09.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:09.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:09.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:09.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:09.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:09.431+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:09.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:09.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:09.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:09.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:09.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:09.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:09.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:09.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:09.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:09.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:09.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:09.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:09.492+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:10.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:10.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:10.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:10.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:10.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:10.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:10.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:10.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:10.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:10.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:10.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:10.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:10.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:10.398+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:10.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:10.398+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:10.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:10.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:10.535+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:10.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:10.536+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:11.120 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:11.001+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52720 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:11.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:11.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:11.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:11.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:11.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:11.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:11.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:11.257+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:11.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:11.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:11.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:11.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:11.504+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:11.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:11.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:11.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:11.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:11.400+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:12.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:12.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:12.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:12.406+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:12.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:12.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:12.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:12.213+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:12.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:12.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:13.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:12.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:13.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:12.551+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:13.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:13.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:13.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:13.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:13.254+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:13.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:13.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:13.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:13.402+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:13.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:13.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:14.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:13.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:14.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:13.546+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:14.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:14.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:14.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:14.205+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:14.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:14.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:14.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:14.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:14.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:15.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:14.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:14.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:14.587+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:15.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:15.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:15.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:15.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:15.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:15.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:15.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:15.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:16.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:15.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:16.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:15.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:16.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:15.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:16.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:15.627+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:16.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:16.008+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52759 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:16.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:16.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:16.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:16.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:16.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:16.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:16.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:16.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:16.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:16.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:16.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:16.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:16.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:16.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:17.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:17.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:17.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:17.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:17.192+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:17.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:17.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:17.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:17.393+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:17.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:17.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:17.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:17.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:18.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:18.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:18.238+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:18.239+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:18.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:18.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:18.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:18.394+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:18.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:19.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:18.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:18.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:19.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:18.657+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:19.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:19.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:19.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:19.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:19.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:19.390+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:19.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:19.391+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:19.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:19.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:19.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:19.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:19.233+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:19.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:19.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:19.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:19.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:19.672+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:20.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:20.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:20.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:20.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:20.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:20.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:20.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:20.235+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:20.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:20.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:20.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:20.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:20.435+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:20.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:20.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:20.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:20.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:20.680+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:21.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:21.014+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52798 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:21.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:21.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:21.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:21.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:21.230+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:21.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:21.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:21.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:21.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:21.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:21.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:21.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:21.406+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:21.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:21.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:21.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:21.660+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:22.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:22.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:22.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:22.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:22.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:22.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:22.253+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:22.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:22.254+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:22.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:22.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:23.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:22.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:22.664+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:23.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:23.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:23.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:23.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:23.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:23.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:23.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:23.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:23.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:23.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:23.342+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:23.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:23.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:23.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:23.711+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:24.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:24.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:24.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:24.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:24.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:24.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:24.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:24.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:24.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:24.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:24.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:24.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:24.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:24.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:25.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:24.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:25.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:24.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:25.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:24.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:24.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:25.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:25.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:25.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:25.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:25.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:25.394+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:25.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:25.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:25.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:25.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:25.222+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:26.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:25.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:26.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:25.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:26.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:25.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:26.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:25.757+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:26.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:26.021+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52837 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:26.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:26.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:26.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:26.212+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:26.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:26.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:26.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:26.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:26.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:26.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:26.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:26.388+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:26.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:26.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:27.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:26.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:27.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:26.724+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:27.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:27.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:27.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:27.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:27.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:27.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:27.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:27.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:27.259+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:27.259+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:27.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:27.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:28.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:27.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:28.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:27.714+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:28.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:28.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:28.433+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:28.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:28.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:28.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:28.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:28.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:28.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:28.218+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:29.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:28.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:29.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:28.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:29.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:28.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:29.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:28.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:29.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:29.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:29.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:29.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:29.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:29.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:29.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:29.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:29.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:29.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:29.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:29.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:29.652+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:29.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:29.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:29.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:29.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:30.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:30.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:30.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:30.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:30.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:30.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:30.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:30.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:30.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:30.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:30.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:30.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:30.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:30.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:30.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:30.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:30.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:30.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:31.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:31.027+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52876 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:31.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:31.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:31.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:31.251+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:31.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:31.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:31.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:31.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:31.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:31.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:31.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:31.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:31.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:32.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:32.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:32.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:32.278+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:32.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:32.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:32.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:32.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:33.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:32.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:33.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:32.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:33.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:33.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:33.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:33.251+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:33.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:33.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:33.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:33.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:33.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:33.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:34.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:33.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:34.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:33.675+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:34.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:34.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:34.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:34.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:34.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:34.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:34.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:34.226+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:34.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:34.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:34.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:34.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:34.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:34.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:34.484+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:34.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:34.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:35.479 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:35.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:35.479 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:35.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:35.479 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:35.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:35.479 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:35.216+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:35.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:35.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:35.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:35.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:35.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:35.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:35.719+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:35.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:35.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:35.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:35.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:35.478+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:36.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:36.034+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52915 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:36.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:36.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:36.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:36.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:36.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:36.218+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:36.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:36.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:36.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:36.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:36.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:36.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:36.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:36.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:36.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:36.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:36.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:36.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:37.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:37.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:37.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:37.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:37.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:37.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:37.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:37.182+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:37.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:37.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:37.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:37.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:37.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:37.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:37.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:37.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:37.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:37.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:37.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:37.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:38.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:38.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:38.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:38.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:38.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:38.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:38.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:38.182+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:38.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:38.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:38.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:38.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:38.691+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:38.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:38.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:38.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:38.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:39.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:39.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:39.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:39.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:39.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:39.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:39.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:39.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:39.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:39.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:39.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:39.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:39.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:39.464+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:39.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:39.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:40.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:39.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:40.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:39.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:40.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:39.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:40.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:39.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:40.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:40.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:40.468 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:40.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:40.468 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:40.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:40.468 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:40.129+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:40.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:40.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:40.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:40.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:40.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:40.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:40.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:40.466+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:40.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:40.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:40.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:40.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:41.460 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:41.041+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52954 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:41.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:41.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:41.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:41.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:41.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:41.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:41.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:41.112+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:41.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:41.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:41.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:41.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:41.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:41.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:41.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:41.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:41.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:42.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:41.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:41.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:42.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:41.780+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:42.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:42.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:42.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:42.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:42.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:42.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:42.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:42.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:42.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:42.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:42.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:42.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:42.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:42.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:42.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:42.415+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:42.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:42.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:43.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:42.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:43.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:42.770+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:43.405 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:43.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:43.405 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:43.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:43.405 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:43.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:43.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:43.107+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:43.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:43.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:43.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:43.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:43.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:43.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:43.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:43.404+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:43.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:43.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:43.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:43.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:44.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:44.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:44.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:44.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:44.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:44.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:44.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:44.094+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:44.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:44.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:44.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:44.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:44.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:45.075 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:44.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:45.075 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:44.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:45.075 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:44.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:45.075 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:44.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:45.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:45.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:45.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:45.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:45.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:45.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:45.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:45.074+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:45.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:45.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:45.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:45.795 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:45.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:46.056 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:46.048+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 52993 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:46.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:45.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:46.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:45.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:46.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:45.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:46.057 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:45.799+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:46.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:46.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:46.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:46.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:46.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:46.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:46.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:46.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:46.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:46.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:46.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:46.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:46.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:46.431+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:46.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:46.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:47.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:46.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:47.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:46.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:47.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:47.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:47.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:47.073+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:47.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:47.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:47.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:47.417+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:47.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:47.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:47.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:47.866+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:48.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:48.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:48.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:48.110+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:48.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:48.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:48.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:48.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:49.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:48.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:48.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:48.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:48.898+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:49.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:49.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:49.091+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:49.091+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:49.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:49.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:49.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:49.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:49.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:49.376+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:50.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:50.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:50.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:50.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:50.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:50.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:50.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:50.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:50.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:49.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:50.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:49.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:50.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:49.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:50.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:49.872+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:50.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:50.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:50.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:50.344+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:51.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:51.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:51.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:51.011+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:50.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:50.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:50.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:50.909+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:51.370 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:51.054+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53032 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:51.368+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:51.368+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:51.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:51.368+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:51.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:51.369+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:51.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:51.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:51.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:51.972+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:51.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:51.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:51.905+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:51.906+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:52.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:52.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:52.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:52.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:52.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:52.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:52.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:52.940+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:52.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:52.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:52.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:52.941+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:53.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:53.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:53.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:53.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:53.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:53.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:53.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:53.907+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:54.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:53.908+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:53.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:53.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:53.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:53.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:54.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:54.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:54.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:54.407+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:54.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:54.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:54.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:54.961+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:54.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:54.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:54.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:55.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:54.931+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:55.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:55.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:55.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:55.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:55.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:56.182 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:04:56.060+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53070 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:04:56.182 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:55.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:56.182 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:55.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:56.182 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:55.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:56.183 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:55.918+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:56.183 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:56.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:56.183 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:56.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:56.183 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:56.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:56.183 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:56.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:56.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:56.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:56.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:56.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:57.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:56.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:57.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:56.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:57.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:56.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:57.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:56.902+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:56.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:56.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:56.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:56.957+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:57.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:57.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:57.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:57.307+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:58.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:57.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:58.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:57.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:58.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:57.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:58.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:57.939+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:58.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:57.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:58.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:57.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:58.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:57.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:58.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:57.896+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:58.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:58.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:58.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:58.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:58.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:58.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:58.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:58.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:58.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:58.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:58.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:04:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:58.969+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:04:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:58.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:58.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:58.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:04:59.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:58.848+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:04:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:59.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:59.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:59.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:04:59.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:04:59.301+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:00.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:59.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:00.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:59.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:00.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:59.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:00.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:04:59.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:00.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:59.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:00.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:59.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:00.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:59.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:00.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:04:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:04:59.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:00.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:00.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:00.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:00.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:01.085 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:01.067+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53110 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:01.085 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:00.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:01.086 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:00.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:01.086 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:00.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:01.086 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:00.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:01.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:00.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:01.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:00.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:01.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:00.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:01.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:00.921+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:01.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:01.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:01.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:01.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:01.315+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:01.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:01.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:01.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:01.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:01.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:01.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:01.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:01.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:02.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:02.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:02.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:02.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:02.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:02.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:03.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:02.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:03.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:02.814+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:02.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:02.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:02.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:02.923+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:03.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:03.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:03.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:03.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:04.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:03.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:03.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:03.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:03.849+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:03.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:03.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:03.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:03.878+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:04.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:04.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:04.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:04.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:04.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:05.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:04.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:05.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:04.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:05.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:04.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:04.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:04.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:05.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:04.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:05.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:04.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:05.787 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:05.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:05.787 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:05.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:05.787 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:05.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:05.787 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:05.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:06.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:05.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:05.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:05.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:06.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:05.786+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:05.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:05.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:05.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:05.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:06.411 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:06.074+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53148 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:06.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:06.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:06.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:06.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:06.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:06.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:06.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:07.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:06.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:07.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:06.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:06.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:06.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:06.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:06.920+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:07.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:07.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:07.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:07.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:07.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:07.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:07.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:07.362+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:07.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:08.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:07.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:07.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:08.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:07.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:07.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:07.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:07.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:08.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:07.908+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:08.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:08.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:08.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:08.402+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:08.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:08.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:08.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:08.890+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:08.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:08.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:08.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:08.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:09.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:09.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:09.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:09.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:09.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:09.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:09.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:09.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:10.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:09.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:10.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:09.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:10.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:09.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:10.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:09.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:09.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:09.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:09.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:10.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:09.874+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:10.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:10.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:10.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:10.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:10.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:11.082 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:10.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:11.082 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:10.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:11.082 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:10.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:11.083 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:10.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:11.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:10.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:11.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:10.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:11.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:10.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:11.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:10.854+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:11.466 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:11.080+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53188 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:11.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:11.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:11.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:11.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:11.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:11.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:11.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:11.465+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:11.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:12.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:11.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:12.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:11.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:12.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:11.781+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:11.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:11.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:11.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:12.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:11.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:12.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:12.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:12.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:12.443+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:12.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:12.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:12.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:12.911+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:12.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:12.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:12.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:12.817+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:13.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:13.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:13.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:13.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:13.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:13.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:13.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:13.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:13.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:13.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:13.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:13.798+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:14.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:14.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:14.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:14.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:14.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:14.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:14.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:14.880+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:14.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:14.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:14.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:14.807+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:15.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:15.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:15.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:15.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:15.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:15.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:15.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:15.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:15.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:16.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:15.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:15.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:16.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:15.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:15.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:15.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:15.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:16.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:15.888+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:16.451 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:16.087+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53226 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:16.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:16.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:16.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:16.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:16.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:16.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:16.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:16.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:17.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:16.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:16.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:16.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:17.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:16.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:16.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:17.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:16.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:17.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:16.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:17.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:16.892+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:17.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:17.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:17.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:17.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:17.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:17.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:17.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:17.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:17.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:17.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:17.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:17.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:18.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:17.884+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:18.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:18.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:18.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:18.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:18.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:18.435+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:18.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:18.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:18.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:18.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:18.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:18.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:18.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:18.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:18.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:18.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:18.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:18.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:18.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:18.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:18.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:18.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:18.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:18.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:19.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:19.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:19.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:19.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:19.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:19.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:19.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:19.394+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:20.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:19.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:20.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:19.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:20.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:19.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:20.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:19.893+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:19.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:19.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:19.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:19.750+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:20.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:20.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:20.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:20.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:20.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:20.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:20.398+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:21.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:20.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:21.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:20.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:21.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:20.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:21.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:20.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:21.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:20.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:21.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:20.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:21.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:20.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:21.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:20.861+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:21.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:21.093+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53267 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:21.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:21.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:21.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:21.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:21.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:21.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:21.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:21.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:21.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:21.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:21.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:21.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:21.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:21.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:21.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:22.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:21.867+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:22.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:22.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:22.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:22.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:22.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:22.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:22.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:22.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:23.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:22.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:23.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:22.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:22.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:23.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:22.847+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:22.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:22.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:22.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:23.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:22.737+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:23.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:23.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:23.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:23.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:23.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:23.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:23.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:23.467+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:23.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:24.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:23.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:23.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:24.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:23.732+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:23.798+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:23.798+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:23.798+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:24.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:23.798+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:24.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:24.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:24.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:24.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:24.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:24.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:24.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:24.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:24.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:24.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:24.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:25.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:24.783+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:25.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:24.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:25.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:24.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:25.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:24.834+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:25.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:24.834+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:25.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:25.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:25.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:25.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:25.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:25.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:25.787 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:25.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:25.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:25.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:26.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:25.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:26.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:25.852+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:25.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:25.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:25.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:26.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:25.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:26.428 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:26.100+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53305 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:26.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:26.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:26.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:26.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:26.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:26.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:26.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:26.873+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:26.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:26.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:26.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:26.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:27.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:27.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:27.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:27.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:27.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:27.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:27.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:27.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:27.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:27.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:27.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:27.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:27.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:27.843+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:28.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:28.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:28.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:28.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:28.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:28.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:28.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:28.842+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:28.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:28.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:28.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:28.852+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:29.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:29.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:29.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:29.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:30.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:29.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:30.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:29.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:30.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:29.867+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:30.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:29.868+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:30.166 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:29.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:30.167 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:29.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:30.167 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:29.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:30.167 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:29.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:30.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:30.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:30.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:30.476+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:31.227 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:31.106+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53345 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:31.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:30.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:31.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:30.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:31.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:30.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:31.228 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:30.901+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:31.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:30.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:31.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:30.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:31.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:30.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:31.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:30.879+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:31.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:31.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:31.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:31.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:31.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:31.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:31.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:31.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:31.887+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:31.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:31.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:31.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:31.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:32.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:32.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:32.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:32.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:32.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:32.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:32.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:32.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:32.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:32.845+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:32.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:32.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:32.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:32.876+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:33.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:33.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:33.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:33.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:33.527+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:33.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:33.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:33.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:33.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:33.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:33.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:33.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:33.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:34.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:34.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:34.880 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:34.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:34.880 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:34.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:34.880 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:34.820+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:34.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:34.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:34.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:34.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:34.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:34.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:34.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:34.571+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:34.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:34.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:34.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:34.878+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:35.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:35.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:36.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:35.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:36.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:35.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:35.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:35.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:35.797+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:36.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:35.798+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:36.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:35.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:36.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:35.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:36.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:35.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:36.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:35.850+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:36.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:36.113+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53383 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:36.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:36.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:36.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:37.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:36.810+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:36.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:36.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:36.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:37.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:36.589+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:37.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:36.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:37.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:36.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:37.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:36.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:37.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:36.770+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:37.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:38.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:37.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:37.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:38.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:37.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:37.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:37.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:37.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:37.779+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:37.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:37.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:37.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:38.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:37.562+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:38.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:38.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:38.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:38.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:38.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:38.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:38.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:38.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:38.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:38.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:38.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:38.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:38.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:38.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:38.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:38.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:38.815+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:39.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:39.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:39.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:39.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:39.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:39.515+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:39.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:39.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:40.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:39.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:39.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:39.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:40.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:39.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:40.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:39.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:40.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:39.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:40.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:39.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:40.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:39.864+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:40.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:40.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:40.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:40.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:40.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:40.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:40.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:40.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:40.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:40.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:41.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:40.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:40.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:40.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:40.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:40.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:40.836+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:41.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:41.119+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53423 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:41.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:41.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:41.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:41.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:41.529+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:41.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:41.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:41.802+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:41.803+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:41.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:41.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:41.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:41.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:42.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:42.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:42.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:42.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:42.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:42.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:42.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:42.773+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:42.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:42.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:42.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:42.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:42.532+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:43.137 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:42.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:43.138 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:42.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:43.138 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:42.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:43.138 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:42.841+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:43.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:43.286+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:43.287348+0000) 2024-02-19T20:05:43.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:43.486+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:43.487775+0000) 2024-02-19T20:05:43.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:43.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:43.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:43.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:43.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:43.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:43.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:43.489+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:43.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:43.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:43.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:43.762+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:43.761+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:43.762443+0000) 2024-02-19T20:05:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:43.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:43.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:43.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:44.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:43.860+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:44.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:44.287+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:44.288443+0000) 2024-02-19T20:05:44.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:44.488+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:44.489707+0000) 2024-02-19T20:05:44.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:44.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:44.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:44.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:44.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:44.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:44.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:44.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:44.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:44.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:44.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:45.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:44.807+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:45.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:44.763+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:44.764373+0000) 2024-02-19T20:05:45.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:44.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:45.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:44.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:45.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:44.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:45.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:44.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:45.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:45.287+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:45.288600+0000) 2024-02-19T20:05:45.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:45.488+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:45.489860+0000) 2024-02-19T20:05:45.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:45.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:45.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:45.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:45.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:45.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:45.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:45.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:45.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:45.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:45.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:46.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:45.792+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:45.763+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:45.764513+0000) 2024-02-19T20:05:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:45.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:45.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:45.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:46.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:45.830+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:46.489 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:46.126+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53462 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:46.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:46.124+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:46.125143+0000) 2024-02-19T20:05:46.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:46.124+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:46.125472+0000) 2024-02-19T20:05:46.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:46.124+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:46.125926+0000) 2024-02-19T20:05:46.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:46.287+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:46.288819+0000) 2024-02-19T20:05:46.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:46.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:46.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:46.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:46.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:46.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:46.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:46.489+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:46.490037+0000) 2024-02-19T20:05:46.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:46.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:46.841+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:46.841+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:46.841+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:47.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:46.842+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:46.763+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:46.764722+0000) 2024-02-19T20:05:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:46.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:46.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:46.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:47.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:46.782+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:47.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:47.290+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:47.291359+0000) 2024-02-19T20:05:47.477 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:47.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:47.477 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:47.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:47.477 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:47.475+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:47.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:47.492+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:47.493591+0000) 2024-02-19T20:05:47.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:47.475+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:47.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:47.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:48.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:47.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:48.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:47.891+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:47.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:47.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:47.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:47.737+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:48.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:47.766+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:47.767275+0000) 2024-02-19T20:05:48.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:48.293+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:48.294734+0000) 2024-02-19T20:05:48.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:48.496+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:48.496999+0000) 2024-02-19T20:05:48.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:48.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:48.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:48.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:48.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:48.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:48.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:48.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:48.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:48.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:48.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:48.911+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:48.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:48.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:48.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:48.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:49.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:48.768+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:48.769230+0000) 2024-02-19T20:05:49.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:49.295+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:49.296717+0000) 2024-02-19T20:05:49.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:49.497+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:49.498254+0000) 2024-02-19T20:05:49.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:49.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:49.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:49.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:49.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:49.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:49.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:49.503+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:49.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:49.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:50.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:49.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:50.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:49.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:49.769+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:49.770361+0000) 2024-02-19T20:05:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:49.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:49.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:49.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:50.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:49.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:50.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:50.295+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:50.296921+0000) 2024-02-19T20:05:50.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:50.497+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:50.498443+0000) 2024-02-19T20:05:50.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:50.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:50.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:50.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:50.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:50.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:50.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:50.513+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:51.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:50.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:50.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:50.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:51.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:50.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:50.769+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:50.770503+0000) 2024-02-19T20:05:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:50.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:50.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:50.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:51.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:50.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:51.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:51.133+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53502 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:51.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:51.130+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:51.131560+0000) 2024-02-19T20:05:51.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:51.130+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:51.131682+0000) 2024-02-19T20:05:51.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:51.130+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:51.131901+0000) 2024-02-19T20:05:51.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:51.296+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:51.297119+0000) 2024-02-19T20:05:51.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:51.497+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:51.498642+0000) 2024-02-19T20:05:51.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:51.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:51.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:51.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:51.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:51.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:51.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:51.500+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:51.769+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:51.770693+0000) 2024-02-19T20:05:52.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:51.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:52.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:51.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:52.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:51.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:52.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:51.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:51.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:51.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:51.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:52.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:51.828+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:52.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:52.298+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:52.299063+0000) 2024-02-19T20:05:52.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:52.499+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:52.500960+0000) 2024-02-19T20:05:52.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:52.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:52.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:52.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:52.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:52.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:52.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:52.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:52.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:52.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:52.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:53.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:52.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:53.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:52.771+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:52.772926+0000) 2024-02-19T20:05:53.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:52.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:53.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:52.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:53.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:52.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:53.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:52.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:53.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:53.300+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:53.301082+0000) 2024-02-19T20:05:53.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:53.502+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:53.503185+0000) 2024-02-19T20:05:53.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:53.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:53.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:53.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:53.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:53.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:53.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:53.560+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:53.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:53.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:53.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:53.891+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:53.774+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:53.775120+0000) 2024-02-19T20:05:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:53.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:53.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:53.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:54.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:53.777+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:54.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:54.301+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:54.302371+0000) 2024-02-19T20:05:54.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:54.503+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:54.504913+0000) 2024-02-19T20:05:54.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:54.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:54.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:54.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:54.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:54.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:54.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:54.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:55.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:54.775+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:54.776452+0000) 2024-02-19T20:05:55.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:54.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:55.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:54.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:54.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:55.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:54.823+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:54.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:54.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:54.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:54.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:55.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:55.301+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:55.302563+0000) 2024-02-19T20:05:55.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:55.504+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:55.505102+0000) 2024-02-19T20:05:55.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:55.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:55.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:55.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:55.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:55.618+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:55.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:55.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:55.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:55.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:55.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:56.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:55.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:55.775+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:55.776653+0000) 2024-02-19T20:05:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:55.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:55.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:55.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:56.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:55.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:56.506 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:05:56.140+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53540 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:05:56.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:56.138+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:56.139031+0000) 2024-02-19T20:05:56.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:56.137+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:56.138853+0000) 2024-02-19T20:05:56.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:56.138+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:56.139421+0000) 2024-02-19T20:05:56.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:56.301+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:56.302818+0000) 2024-02-19T20:05:56.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:56.504+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:56.505286+0000) 2024-02-19T20:05:56.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:56.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:56.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:56.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:56.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:56.627+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:56.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:56.628+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:56.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:56.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:56.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:57.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:56.901+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:56.775+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:56.776882+0000) 2024-02-19T20:05:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:56.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:56.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:56.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:57.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:56.786+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:57.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:57.304+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:57.305673+0000) 2024-02-19T20:05:57.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:57.506+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:57.507891+0000) 2024-02-19T20:05:57.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:57.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:57.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:57.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:57.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:57.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:57.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:57.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:57.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:57.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:57.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:57.892+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:58.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:57.778+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:57.779210+0000) 2024-02-19T20:05:58.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:57.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:58.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:57.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:58.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:57.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:58.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:57.792+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:58.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:58.308+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:58.309323+0000) 2024-02-19T20:05:58.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:58.509+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:58.510449+0000) 2024-02-19T20:05:58.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:58.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:58.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:58.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:58.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:58.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:58.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:58.682+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:58.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:58.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:58.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:05:59.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:58.938+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:05:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:58.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:58.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:58.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:58.781+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:58.782667+0000) 2024-02-19T20:05:59.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:58.782+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:59.309+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:59.310636+0000) 2024-02-19T20:05:59.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:59.511+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:59.512162+0000) 2024-02-19T20:05:59.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:59.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:59.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:59.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:59.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:59.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:05:59.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:05:59.730+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:05:59.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:59.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:59.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:59.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:59.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:59.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:05:59.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:59.743+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:00.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:05:59.783+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:05:59.784235+0000) 2024-02-19T20:06:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:59.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:00.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:59.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:00.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:59.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:00.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:05:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:05:59.891+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:00.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:00.309+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:00.310962+0000) 2024-02-19T20:06:00.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:00.511+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:00.512353+0000) 2024-02-19T20:06:00.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:00.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:00.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:00.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:00.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:00.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:00.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:00.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:00.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:00.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:00.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:00.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:00.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:00.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:00.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:00.684+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:01.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:00.783+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:00.784493+0000) 2024-02-19T20:06:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:00.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:00.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:01.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:00.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:01.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:00.913+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:01.425 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:01.147+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53580 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:01.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:01.144+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:01.145228+0000) 2024-02-19T20:06:01.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:01.144+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:01.145989+0000) 2024-02-19T20:06:01.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:01.144+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:01.145934+0000) 2024-02-19T20:06:01.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:01.310+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:01.311154+0000) 2024-02-19T20:06:01.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:01.511+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:01.512569+0000) 2024-02-19T20:06:01.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:01.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:01.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:01.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:01.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:01.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:01.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:01.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:01.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:01.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:01.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:02.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:01.891+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:01.758+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:02.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:01.783+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:01.784718+0000) 2024-02-19T20:06:02.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:02.312+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:02.313791+0000) 2024-02-19T20:06:02.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:02.515+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:02.516118+0000) 2024-02-19T20:06:02.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:02.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:02.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:02.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:02.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:02.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:02.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:02.654+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:03.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:02.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:03.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:02.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:03.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:02.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:02.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:03.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:02.785+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:02.786686+0000) 2024-02-19T20:06:03.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:02.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:03.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:02.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:03.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:02.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:03.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:02.876+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:03.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:03.315+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:03.316244+0000) 2024-02-19T20:06:03.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:03.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:03.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:03.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:03.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:03.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:03.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:03.741+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:03.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:03.517+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:03.518368+0000) 2024-02-19T20:06:03.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:03.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:03.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:03.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:03.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:03.605+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:03.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:03.605+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:04.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:03.787+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:03.788666+0000) 2024-02-19T20:06:04.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:03.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:04.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:03.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:04.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:03.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:04.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:03.834+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:04.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:04.316+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:04.317738+0000) 2024-02-19T20:06:04.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:04.518+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:04.519654+0000) 2024-02-19T20:06:04.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:04.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:04.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:04.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:04.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:04.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:04.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:04.757+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:04.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:04.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:04.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:04.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:04.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:04.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:04.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:04.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:05.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:04.788+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:04.789881+0000) 2024-02-19T20:06:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:04.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:05.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:04.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:04.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:05.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:04.813+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:05.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:05.316+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:05.317904+0000) 2024-02-19T20:06:05.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:05.518+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:05.519830+0000) 2024-02-19T20:06:05.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:05.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:05.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:05.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:05.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:05.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:05.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:05.732+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:05.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:05.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:05.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:05.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:05.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:05.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:05.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:06.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:05.789+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:05.790086+0000) 2024-02-19T20:06:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:05.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:05.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:05.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:06.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:05.798+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:06.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:06.150+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:06.151864+0000) 2024-02-19T20:06:06.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:06.151+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:06.152483+0000) 2024-02-19T20:06:06.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:06.317+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:06.318067+0000) 2024-02-19T20:06:06.521 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:06.154+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53618 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:06.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:06.151+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:06.152745+0000) 2024-02-19T20:06:06.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:06.519+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:06.520058+0000) 2024-02-19T20:06:06.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:06.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:06.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:06.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:06.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:06.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:06.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:06.750+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:06.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:06.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:06.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:06.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:06.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:06.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:06.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:06.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:06.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:06.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:06.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:07.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:06.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:07.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:06.789+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:06.790273+0000) 2024-02-19T20:06:07.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:07.319+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:07.320645+0000) 2024-02-19T20:06:07.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:07.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:07.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:07.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:07.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:07.521+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:07.522557+0000) 2024-02-19T20:06:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:07.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:07.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:07.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:08.093 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:07.791+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:07.792855+0000) 2024-02-19T20:06:08.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:07.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:08.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:07.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:08.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:07.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:08.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:07.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:08.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:07.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:08.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:07.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:08.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:07.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:08.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:07.792+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:08.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:08.322+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:08.323176+0000) 2024-02-19T20:06:08.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:08.523+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:08.524634+0000) 2024-02-19T20:06:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:08.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:08.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:08.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:08.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:08.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:09.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:08.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:09.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:08.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:09.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:08.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:09.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:08.826+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:09.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:08.795+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:08.796482+0000) 2024-02-19T20:06:09.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:08.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:09.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:08.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:09.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:08.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:09.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:08.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:09.323+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:09.324533+0000) 2024-02-19T20:06:09.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:09.524+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:09.525949+0000) 2024-02-19T20:06:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:09.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:09.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:09.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:09.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:10.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:09.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:10.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:09.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:10.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:09.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:10.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:09.864+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:10.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:09.796+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:09.797694+0000) 2024-02-19T20:06:10.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:09.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:10.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:09.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:10.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:09.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:10.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:09.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:10.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:10.323+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:10.324713+0000) 2024-02-19T20:06:10.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:10.525+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:10.526178+0000) 2024-02-19T20:06:10.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:10.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:10.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:10.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:10.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:10.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:10.785 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:10.570+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:10.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:10.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:10.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:11.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:10.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:10.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:10.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:11.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:10.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:11.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:10.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:11.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:10.796+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:10.797923+0000) 2024-02-19T20:06:11.527 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:11.160+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53658 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:11.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:11.160+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:11.161221+0000) 2024-02-19T20:06:11.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:11.159+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:11.160720+0000) 2024-02-19T20:06:11.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:11.159+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:11.160973+0000) 2024-02-19T20:06:11.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:11.323+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:11.324917+0000) 2024-02-19T20:06:11.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:11.525+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:11.526345+0000) 2024-02-19T20:06:11.808 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:11.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:11.808 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:11.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:11.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:11.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:11.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:11.583+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:12.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:11.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:12.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:11.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:12.094 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:11.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:12.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:11.866+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:12.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:11.797+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:11.798162+0000) 2024-02-19T20:06:12.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:11.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:12.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:11.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:12.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:11.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:12.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:11.819+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:12.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:12.326+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:12.327064+0000) 2024-02-19T20:06:12.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:12.528+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:12.529229+0000) 2024-02-19T20:06:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:12.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:12.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:12.625+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:12.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:12.626+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:13.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:12.800+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:12.801643+0000) 2024-02-19T20:06:13.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:12.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:13.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:12.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:13.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:12.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:13.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:12.824+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:13.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:12.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:13.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:12.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:13.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:12.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:13.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:12.866+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:13.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:13.329+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:13.330668+0000) 2024-02-19T20:06:13.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:13.530+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:13.531490+0000) 2024-02-19T20:06:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:13.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:13.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:13.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:13.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:13.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:13.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:14.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:13.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:14.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:13.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:14.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:13.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:14.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:13.915+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:14.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:13.804+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:13.805074+0000) 2024-02-19T20:06:14.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:13.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:14.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:13.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:14.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:13.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:14.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:13.824+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:14.533 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:14.331+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:14.332759+0000) 2024-02-19T20:06:14.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:14.531+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:14.532885+0000) 2024-02-19T20:06:14.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:14.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:14.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:14.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:14.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:14.647+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:14.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:14.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:14.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:14.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:14.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:15.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:14.806+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:14.806233+0000) 2024-02-19T20:06:15.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:14.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:15.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:14.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:15.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:14.919+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:15.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:14.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:15.533 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:15.332+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:15.332960+0000) 2024-02-19T20:06:15.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:15.533+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:15.533081+0000) 2024-02-19T20:06:15.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:15.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:15.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:15.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:15.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:15.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:15.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:15.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:15.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:15.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:15.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:15.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:15.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:15.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:15.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:16.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:15.806+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:15.806440+0000) 2024-02-19T20:06:16.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:15.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:16.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:15.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:16.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:15.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:16.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:15.923+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:16.533 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:16.167+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53696 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:16.534 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:16.165+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:16.166044+0000) 2024-02-19T20:06:16.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:16.165+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:16.166603+0000) 2024-02-19T20:06:16.534 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:16.165+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:16.166893+0000) 2024-02-19T20:06:16.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:16.332+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:16.333151+0000) 2024-02-19T20:06:16.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:16.532+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:16.533279+0000) 2024-02-19T20:06:16.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:16.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:16.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:16.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:16.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:16.740+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:16.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:16.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:16.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:16.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:16.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:16.721+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:16.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:16.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:17.094 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:16.805+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:16.806652+0000) 2024-02-19T20:06:17.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:16.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:17.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:16.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:17.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:16.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:17.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:16.916+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:17.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:17.334+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:17.335127+0000) 2024-02-19T20:06:17.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:17.535+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:17.536592+0000) 2024-02-19T20:06:17.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:17.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:17.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:17.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:17.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:17.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:17.788 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:17.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:17.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:18.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:17.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:17.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:17.787+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:17.807+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:17.808657+0000) 2024-02-19T20:06:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:17.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:17.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:17.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:18.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:17.884+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:18.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:18.336+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:18.337110+0000) 2024-02-19T20:06:18.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:18.538+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:18.539997+0000) 2024-02-19T20:06:18.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:18.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:18.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:18.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:18.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:18.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:18.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:18.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:18.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:18.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:18.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:18.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:18.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:18.710+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:19.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:18.809+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:18.810836+0000) 2024-02-19T20:06:19.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:18.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:19.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:18.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:19.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:18.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:19.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:18.916+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:19.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:19.337+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:19.338471+0000) 2024-02-19T20:06:19.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:19.540+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:19.541944+0000) 2024-02-19T20:06:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:19.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:19.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:19.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:19.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:20.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:19.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:20.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:19.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:20.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:19.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:20.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:19.956+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:20.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:19.811+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:19.812059+0000) 2024-02-19T20:06:20.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:19.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:20.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:19.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:20.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:19.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:20.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:19.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:20.337+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:20.338657+0000) 2024-02-19T20:06:20.812 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:20.541+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:20.542110+0000) 2024-02-19T20:06:20.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:20.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:20.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:20.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:20.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:20.696+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:20.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:20.696+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:20.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:20.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:20.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:20.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:20.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:20.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:20.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:20.786+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:21.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:20.811+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:20.812129+0000) 2024-02-19T20:06:21.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:20.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:21.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:20.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:21.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:20.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:21.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:20.913+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:21.171+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:21.173002+0000) 2024-02-19T20:06:21.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:21.174+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53736 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:21.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:21.172+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:21.173581+0000) 2024-02-19T20:06:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:21.171+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:21.172706+0000) 2024-02-19T20:06:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:21.337+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:21.338863+0000) 2024-02-19T20:06:21.807 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:21.541+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:21.542282+0000) 2024-02-19T20:06:21.807 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:21.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:21.807 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:21.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:21.807 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:21.733+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:21.808 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:21.733+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:22.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:21.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:22.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:21.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:22.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:21.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:22.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:21.901+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:22.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:21.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:22.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:21.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:22.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:21.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:22.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:21.806+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:22.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:21.811+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:21.812346+0000) 2024-02-19T20:06:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:22.341+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:22.342163+0000) 2024-02-19T20:06:22.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:22.543+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:22.544839+0000) 2024-02-19T20:06:22.815 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:22.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:22.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:22.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:22.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:22.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:22.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:22.708+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:22.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:22.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:22.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:22.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:22.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:22.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:22.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:22.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:23.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:22.813+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:22.814868+0000) 2024-02-19T20:06:23.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:22.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:23.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:22.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:23.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:22.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:23.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:22.869+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:23.343+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:23.344727+0000) 2024-02-19T20:06:23.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:23.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:23.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:23.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:23.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:23.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:23.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:23.786+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:23.818 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:23.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:23.819 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:23.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:23.819 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:23.709+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:23.819 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:23.710+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:23.819 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:23.545+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:23.546997+0000) 2024-02-19T20:06:24.095 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:23.816+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:23.817392+0000) 2024-02-19T20:06:24.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:23.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:24.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:23.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:24.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:23.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:24.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:23.866+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:24.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:24.344+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:24.345993+0000) 2024-02-19T20:06:24.614 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:24.547+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:24.548725+0000) 2024-02-19T20:06:24.891 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:24.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:24.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:24.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:24.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:24.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:24.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:24.723+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:24.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:24.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:24.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:24.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:24.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:24.820+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:24.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:24.821+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:24.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:24.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:24.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:24.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:24.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:24.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:24.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:24.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:24.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:24.818+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:24.819443+0000) 2024-02-19T20:06:25.614 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:25.548+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:25.549008+0000) 2024-02-19T20:06:25.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:25.345+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:25.346155+0000) 2024-02-19T20:06:25.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:25.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:25.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:25.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:25.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:25.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:25.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:25.692+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:25.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:25.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:25.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:25.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:25.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:25.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:25.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:25.806+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:25.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:25.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:25.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:25.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:25.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:25.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:25.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:25.766+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:25.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:25.818+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:25.819628+0000) 2024-02-19T20:06:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:26.178+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:26.179232+0000) 2024-02-19T20:06:26.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:26.180+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53774 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:26.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:26.178+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:26.179336+0000) 2024-02-19T20:06:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:26.178+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:26.179937+0000) 2024-02-19T20:06:26.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:26.345+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:26.346308+0000) 2024-02-19T20:06:26.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:26.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:26.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:26.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:26.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:26.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:26.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:26.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:26.821 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:26.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:26.821 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:26.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:26.821 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:26.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:26.821 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:26.663+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:26.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:26.548+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:26.549279+0000) 2024-02-19T20:06:26.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:26.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:26.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:26.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:26.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:26.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:26.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:26.797+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:27.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:26.818+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:26.819801+0000) 2024-02-19T20:06:27.614 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:27.550+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:27.551357+0000) 2024-02-19T20:06:27.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:27.347+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:27.348384+0000) 2024-02-19T20:06:27.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:27.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:27.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:27.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:27.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:27.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:27.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:27.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:27.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:27.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:27.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:27.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:27.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:27.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:27.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:27.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:27.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:27.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:27.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:27.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:27.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:27.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:27.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:27.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:27.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:27.821+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:27.822317+0000) 2024-02-19T20:06:28.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:28.552+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:28.553512+0000) 2024-02-19T20:06:28.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:28.350+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:28.351071+0000) 2024-02-19T20:06:28.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:28.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:28.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:28.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:28.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:28.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:28.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:28.629+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:28.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:28.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:28.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:28.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:28.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:28.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:28.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:28.765+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:28.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:28.823+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:28.824355+0000) 2024-02-19T20:06:28.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:28.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:28.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:28.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:28.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:28.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:28.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:28.839+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:29.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:29.554+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:29.555584+0000) 2024-02-19T20:06:29.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:29.351+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:29.352267+0000) 2024-02-19T20:06:29.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:29.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:29.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:29.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:29.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:29.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:29.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:29.843+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:29.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:29.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:29.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:29.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:29.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:29.781+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:29.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:29.782+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:29.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:29.824+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:29.825847+0000) 2024-02-19T20:06:29.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:29.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:29.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:29.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:29.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:29.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:29.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:29.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:30.351+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:30.352451+0000) 2024-02-19T20:06:30.827 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:30.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:30.827 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:30.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:30.828 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:30.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:30.828 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:30.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:30.828 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:30.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:30.828 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:30.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:30.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:30.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:30.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:30.687+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:30.829 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:30.554+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:30.555775+0000) 2024-02-19T20:06:31.096 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:30.825+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:30.826056+0000) 2024-02-19T20:06:31.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:30.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:31.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:30.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:31.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:30.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:31.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:30.851+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:31.347 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:31.187+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53814 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:31.348 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:31.185+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:31.186245+0000) 2024-02-19T20:06:31.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:31.184+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:31.185543+0000) 2024-02-19T20:06:31.348 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:31.185+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:31.186104+0000) 2024-02-19T20:06:31.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:31.351+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:31.352649+0000) 2024-02-19T20:06:31.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:31.554+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:31.556000+0000) 2024-02-19T20:06:31.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:31.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:31.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:31.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:31.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:31.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:31.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:31.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:31.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:31.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:31.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:31.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:31.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:31.805+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:31.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:31.806+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:31.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:31.825+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:31.826282+0000) 2024-02-19T20:06:32.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:31.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:32.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:31.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:32.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:31.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:32.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:31.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:32.353+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:32.354670+0000) 2024-02-19T20:06:32.831 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:32.557+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:32.558577+0000) 2024-02-19T20:06:32.831 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:32.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:32.832 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:32.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:32.833 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:32.644+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:32.834 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:32.645+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:33.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:32.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:33.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:32.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:33.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:32.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:33.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:32.907+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:33.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:32.828+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:32.829068+0000) 2024-02-19T20:06:33.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:32.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:33.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:32.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:33.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:32.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:33.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:32.852+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:33.612 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:33.356+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:33.357194+0000) 2024-02-19T20:06:33.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:33.560+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:33.561156+0000) 2024-02-19T20:06:33.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:33.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:33.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:33.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:33.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:33.611+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:33.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:33.611+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:33.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:33.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:33.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:33.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:33.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:33.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:33.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:33.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:33.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:33.830+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:33.831603+0000) 2024-02-19T20:06:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:33.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:33.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:33.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:33.929+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:34.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:34.561+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:34.562527+0000) 2024-02-19T20:06:34.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:34.357+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:34.358428+0000) 2024-02-19T20:06:34.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:34.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:34.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:34.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:34.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:34.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:34.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:34.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:34.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:34.831+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:34.832891+0000) 2024-02-19T20:06:34.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:34.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:34.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:34.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:34.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:34.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:34.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:34.857+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:34.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:34.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:34.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:34.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:35.615 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:35.561+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:35.562685+0000) 2024-02-19T20:06:35.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:35.357+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:35.358613+0000) 2024-02-19T20:06:35.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:35.831+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:35.832994+0000) 2024-02-19T20:06:35.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:35.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:35.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:35.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:35.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:35.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:35.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:35.850+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:35.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:35.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:35.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:35.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:35.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:35.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:35.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:35.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:36.190 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:35.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:36.191 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:35.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:36.191 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:35.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:36.191 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:35.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:36.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:36.194+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53852 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:36.191+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:36.192793+0000) 2024-02-19T20:06:36.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:36.192+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:36.193264+0000) 2024-02-19T20:06:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:36.191+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:36.192774+0000) 2024-02-19T20:06:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:36.357+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:36.358831+0000) 2024-02-19T20:06:36.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:36.561+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:36.562888+0000) 2024-02-19T20:06:36.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:36.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:36.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:36.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:36.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:36.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:36.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:36.667+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:37.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:36.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:37.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:36.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:37.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:36.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:37.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:36.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:37.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:36.832+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:36.833189+0000) 2024-02-19T20:06:37.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:36.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:37.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:36.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:37.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:36.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:37.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:36.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:37.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:37.359+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:37.360936+0000) 2024-02-19T20:06:37.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:37.564+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:37.565407+0000) 2024-02-19T20:06:37.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:37.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:37.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:37.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:37.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:37.711+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:37.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:37.711+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:37.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:37.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:37.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:37.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:37.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:37.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:37.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:37.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:37.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:37.834+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:37.835338+0000) 2024-02-19T20:06:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:37.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:37.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:38.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:37.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:38.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:37.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:38.568 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:38.363+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:38.364518+0000) 2024-02-19T20:06:38.838 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:38.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:38.838 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:38.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:38.838 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:38.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:38.838 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:38.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:38.838 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:38.567+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:38.568078+0000) 2024-02-19T20:06:38.839 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:38.839 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:38.839 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:38.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:38.839 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:38.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:39.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:38.836+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:38.837521+0000) 2024-02-19T20:06:39.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:38.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:39.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:38.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:39.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:38.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:39.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:38.902+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:39.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:39.569+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:39.570201+0000) 2024-02-19T20:06:39.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:39.364+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:39.365706+0000) 2024-02-19T20:06:39.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:39.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:39.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:39.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:39.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:39.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:39.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:39.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:39.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:39.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:39.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:39.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:39.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:39.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:39.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:39.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:39.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:39.838+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:39.839547+0000) 2024-02-19T20:06:40.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:39.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:40.170 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:39.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:40.170 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:39.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:40.170 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:39.876+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:40.364+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:40.365936+0000) 2024-02-19T20:06:40.840 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:40.569+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:40.570388+0000) 2024-02-19T20:06:40.840 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:40.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:40.841 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:40.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:40.841 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:40.663+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:40.841 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:40.664+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:40.841 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:40.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:40.841 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:40.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:40.841 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:40.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:40.841 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:40.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:41.097 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:40.838+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:40.839740+0000) 2024-02-19T20:06:41.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:40.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:41.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:40.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:41.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:40.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:41.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:40.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:41.354 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:41.200+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53892 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:41.354 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:41.199+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:41.200098+0000) 2024-02-19T20:06:41.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:41.198+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:41.200014+0000) 2024-02-19T20:06:41.355 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:41.198+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:41.199185+0000) 2024-02-19T20:06:41.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:41.569+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:41.570598+0000) 2024-02-19T20:06:41.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:41.365+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:41.366128+0000) 2024-02-19T20:06:41.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:41.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:41.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:41.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:41.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:41.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:41.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:41.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:41.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:41.838+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:41.839908+0000) 2024-02-19T20:06:41.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:41.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:41.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:41.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:41.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:41.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:41.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:41.676+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:41.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:41.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:41.930+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:41.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:42.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:42.571+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:42.572730+0000) 2024-02-19T20:06:42.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:42.368+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:42.369266+0000) 2024-02-19T20:06:42.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:42.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:42.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:42.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:42.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:42.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:42.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:42.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:42.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:42.840+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:42.842006+0000) 2024-02-19T20:06:42.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:42.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:42.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:42.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:42.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:42.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:42.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:42.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:42.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:42.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:42.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:42.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:43.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:43.574+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:43.575350+0000) 2024-02-19T20:06:43.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:43.370+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:43.371290+0000) 2024-02-19T20:06:43.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:43.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:43.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:43.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:43.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:43.631+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:43.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:43.632+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:43.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:43.843+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:43.844107+0000) 2024-02-19T20:06:43.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:43.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:43.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:43.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:43.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:43.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:43.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:43.857+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:43.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:43.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:43.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:43.910+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:44.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:44.576+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:44.577199+0000) 2024-02-19T20:06:44.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:44.371+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:44.372834+0000) 2024-02-19T20:06:44.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:44.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:44.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:44.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:44.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:44.665+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:44.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:44.666+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:44.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:44.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:44.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:44.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:44.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:44.837+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:44.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:44.837+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:44.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:44.844+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:44.845336+0000) 2024-02-19T20:06:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:44.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:44.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:44.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:44.895+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:45.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:45.371+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:45.372988+0000) 2024-02-19T20:06:45.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:45.576+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:45.577355+0000) 2024-02-19T20:06:45.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:45.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:45.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:45.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:45.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:45.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:45.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:45.714+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:45.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:45.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:45.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:45.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:45.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:45.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:45.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:45.843+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:45.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:45.844+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:45.845546+0000) 2024-02-19T20:06:46.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:45.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:46.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:45.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:46.204 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:45.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:46.204 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:45.879+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:46.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:46.204+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:46.205801+0000) 2024-02-19T20:06:46.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:46.207+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53930 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:46.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:46.205+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:46.206478+0000) 2024-02-19T20:06:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:46.205+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:46.206850+0000) 2024-02-19T20:06:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:46.372+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:46.373170+0000) 2024-02-19T20:06:46.829 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:46.576+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:46.577537+0000) 2024-02-19T20:06:46.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:46.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:46.830 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:46.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:46.830 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:46.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:46.830 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:46.684+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:47.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:46.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:47.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:46.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:47.099 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:46.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:47.099 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:46.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:47.099 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:46.844+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:46.845756+0000) 2024-02-19T20:06:47.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:46.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:47.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:46.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:47.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:46.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:47.100 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:46.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:47.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:47.579+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:47.580136+0000) 2024-02-19T20:06:47.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:47.374+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:47.375202+0000) 2024-02-19T20:06:47.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:47.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:47.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:47.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:47.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:47.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:47.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:47.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:47.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:47.847+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:47.848175+0000) 2024-02-19T20:06:47.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:47.674+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:47.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:47.674+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:47.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:47.674+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:47.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:47.675+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:47.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:47.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:47.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:47.885+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:48.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:48.581+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:48.582290+0000) 2024-02-19T20:06:48.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:48.376+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:48.377242+0000) 2024-02-19T20:06:48.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:48.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:48.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:48.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:48.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:48.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:48.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:48.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:48.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:48.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:48.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:48.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:48.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:48.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:48.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:48.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:48.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:48.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:48.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:48.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:48.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:48.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:48.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:48.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:48.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:48.849+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:48.850773+0000) 2024-02-19T20:06:49.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:49.377+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:49.378525+0000) 2024-02-19T20:06:49.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:49.582+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:49.583858+0000) 2024-02-19T20:06:49.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:49.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:49.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:49.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:49.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:49.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:49.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:49.691+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:49.896 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:49.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:49.896 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:49.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:49.896 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:49.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:49.896 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:49.855+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:49.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:49.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:49.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:49.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:49.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:49.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:49.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:49.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:49.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:49.851+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:49.852106+0000) 2024-02-19T20:06:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:50.377+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:50.378683+0000) 2024-02-19T20:06:50.852 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:50.583+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:50.584097+0000) 2024-02-19T20:06:50.853 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:50.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:50.853 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:50.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:50.853 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:50.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:50.853 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:50.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:50.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:50.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:50.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:50.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:50.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:50.816+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:50.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:50.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:51.210 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:50.851+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:50.852282+0000) 2024-02-19T20:06:51.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:50.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:51.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:50.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:51.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:50.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:51.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:50.882+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:51.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:51.214+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 53970 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:51.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:51.212+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:51.213100+0000) 2024-02-19T20:06:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:51.212+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:51.213238+0000) 2024-02-19T20:06:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:51.377+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:51.378890+0000) 2024-02-19T20:06:51.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:51.212+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:51.213354+0000) 2024-02-19T20:06:51.853 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:51.583+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:51.584274+0000) 2024-02-19T20:06:51.853 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:51.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:51.853 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:51.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:51.853 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:51.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:51.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:51.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:51.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:51.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:51.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:51.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:51.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:51.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:51.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:51.751+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:51.851+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:51.852469+0000) 2024-02-19T20:06:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:51.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:51.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:51.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:51.896+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:52.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:52.585+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:52.586357+0000) 2024-02-19T20:06:52.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:52.380+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:52.381042+0000) 2024-02-19T20:06:52.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:52.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:52.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:52.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:52.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:52.718+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:52.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:52.718+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:52.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:52.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:52.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:52.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:52.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:52.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:52.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:52.791+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:52.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:52.854+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:52.855135+0000) 2024-02-19T20:06:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:52.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:52.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:52.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:52.916+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:53.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:53.382+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:53.383658+0000) 2024-02-19T20:06:53.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:53.588+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:53.589974+0000) 2024-02-19T20:06:53.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:53.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:53.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:53.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:53.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:53.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:53.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:53.741+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:53.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:53.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:53.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:53.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:53.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:53.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:53.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:53.773+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:53.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:53.856+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:53.857961+0000) 2024-02-19T20:06:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:53.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:53.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:53.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:53.897+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:54.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:54.384+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:54.385060+0000) 2024-02-19T20:06:54.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:54.590+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:54.591362+0000) 2024-02-19T20:06:54.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:54.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:54.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:54.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:54.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:54.759+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:54.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:54.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:54.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:54.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:54.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:54.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:54.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:54.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:54.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:54.804+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:54.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:54.858+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:54.859266+0000) 2024-02-19T20:06:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:54.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:54.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:54.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:54.946+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:55.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:55.384+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:55.385259+0000) 2024-02-19T20:06:55.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:55.590+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:55.591531+0000) 2024-02-19T20:06:55.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:55.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:55.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:55.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:55.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:55.728+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:55.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:55.728+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:55.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:55.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:55.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:55.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:55.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:55.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:55.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:55.817+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:55.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:55.858+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:55.859454+0000) 2024-02-19T20:06:56.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:55.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:56.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:55.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:56.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:55.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:56.217 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:55.981+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:56.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:06:56.221+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54008 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:06:56.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:56.218+0000 7f49ebb90700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:56.219704+0000) 2024-02-19T20:06:56.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:56.219+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:56.220818+0000) 2024-02-19T20:06:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:56.219+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:56.220438+0000) 2024-02-19T20:06:56.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:56.384+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:56.385467+0000) 2024-02-19T20:06:56.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:56.590+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:56.591716+0000) 2024-02-19T20:06:56.860 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:56.776+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:56.861 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:56.776+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:56.861 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:56.776+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:56.861 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:56.777+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:56.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:56.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:56.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:56.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:56.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:56.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:56.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:56.813+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:57.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:56.858+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:56.859651+0000) 2024-02-19T20:06:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:57.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:57.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:57.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:57.014+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:57.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:57.593+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:57.594245+0000) 2024-02-19T20:06:57.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:57.386+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:57.387483+0000) 2024-02-19T20:06:57.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:57.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:57.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:57.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:57.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:57.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:57.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:57.750+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:57.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:57.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:57.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:57.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:57.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:57.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:57.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:57.802+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:57.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:57.861+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:57.862093+0000) 2024-02-19T20:06:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:58.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.595+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:58.596185+0000) 2024-02-19T20:06:58.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:58.388+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:58.389688+0000) 2024-02-19T20:06:58.896 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:58.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:58.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:58.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:58.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:58.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:58.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:58.760+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:58.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:58.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:58.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:58.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:58.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:58.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:58.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:58.792+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:58.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:58.863+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:58.864058+0000) 2024-02-19T20:06:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:06:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:58.968+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:06:59.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:59.596+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:59.597576+0000) 2024-02-19T20:06:59.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:59.390+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:59.391151+0000) 2024-02-19T20:06:59.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:59.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:59.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:59.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:59.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:59.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:06:59.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:59.790+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:06:59.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:06:59.865+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:06:59.866284+0000) 2024-02-19T20:06:59.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:59.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:59.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:59.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:59.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:59.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:06:59.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:06:59.809+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:00.171 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:59.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:00.171 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:59.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:00.171 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:59.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:00.171 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:06:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:06:59.922+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:00.390+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:00.391309+0000) 2024-02-19T20:07:00.866 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:00.596+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:00.597795+0000) 2024-02-19T20:07:00.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:00.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:00.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:00.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:00.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:00.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:00.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:00.773+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:00.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:00.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:00.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:00.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:00.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:00.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:00.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:00.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:01.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:00.865+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:00.866443+0000) 2024-02-19T20:07:01.224 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:00.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:01.224 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:00.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:01.224 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:00.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:01.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:00.874+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:01.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:01.228+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54048 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:01.225+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:01.226710+0000) 2024-02-19T20:07:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:01.226+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:01.227235+0000) 2024-02-19T20:07:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:01.390+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:01.391517+0000) 2024-02-19T20:07:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:01.225+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:01.226601+0000) 2024-02-19T20:07:01.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:01.598+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:01.598048+0000) 2024-02-19T20:07:01.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:01.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:01.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:01.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:01.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:01.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:01.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:01.759+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:01.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:01.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:01.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:01.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:01.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:01.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:01.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:01.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:02.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:01.866+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:01.866706+0000) 2024-02-19T20:07:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:01.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:01.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:01.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:01.889+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:02.619 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:02.600+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:02.601841+0000) 2024-02-19T20:07:02.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:02.394+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:02.395168+0000) 2024-02-19T20:07:02.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:02.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:02.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:02.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:02.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:02.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:02.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:02.808+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:02.870 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:02.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:02.870 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:02.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:02.870 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:02.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:02.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:02.782+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:02.869+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:02.869156+0000) 2024-02-19T20:07:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:02.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:02.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:02.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:02.882+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:03.619 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:03.604+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:03.605448+0000) 2024-02-19T20:07:03.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:03.397+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:03.397384+0000) 2024-02-19T20:07:03.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:03.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:03.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:03.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:03.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:03.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:03.873 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:03.818+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:03.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:03.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:03.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:03.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:03.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:03.780+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:03.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:03.781+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:04.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:03.870+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:03.871616+0000) 2024-02-19T20:07:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:03.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:03.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:03.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:03.908+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:04.619 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:04.606+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:04.607682+0000) 2024-02-19T20:07:04.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:04.397+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:04.398719+0000) 2024-02-19T20:07:04.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:04.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:04.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:04.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:04.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:04.800+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:04.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:04.801+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:04.874 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:04.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:04.874 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:04.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:04.875 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:04.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:04.875 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:04.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:04.872+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:04.873137+0000) 2024-02-19T20:07:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:04.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:04.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:04.892+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:04.893+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:05.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:05.398+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:05.399046+0000) 2024-02-19T20:07:05.619 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:05.606+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:05.607874+0000) 2024-02-19T20:07:05.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:05.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:05.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:05.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:05.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:05.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:05.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:05.781+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:05.874 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:05.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:05.875 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:05.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:05.875 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:05.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:32.512420+0000 (oldest deadline 2024-02-19T19:56:52.512420+0000) 2024-02-19T20:07:05.875 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:05.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:06.231 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:05.872+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:05.873401+0000) 2024-02-19T20:07:06.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:05.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:06.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:05.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:06.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:05.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:06.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:05.943+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:06.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:06.234+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54086 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:06.232+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:06.233993+0000) 2024-02-19T20:07:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:06.232+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:06.233804+0000) 2024-02-19T20:07:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:06.398+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:06.399289+0000) 2024-02-19T20:07:06.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:06.232+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:06.233665+0000) 2024-02-19T20:07:06.874 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:06.780+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:06.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:06.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:06.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:06.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:06.875 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:06.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:06.875 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:06.753+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:06.875 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:06.607+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:06.608066+0000) 2024-02-19T20:07:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:06.872+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:06.873581+0000) 2024-02-19T20:07:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:06.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:06.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:06.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:06.977+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:07.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:07.400+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:07.401347+0000) 2024-02-19T20:07:07.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:07.610+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:07.611573+0000) 2024-02-19T20:07:07.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:07.781+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:07.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:07.875+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:07.876257+0000) 2024-02-19T20:07:07.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:07.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:07.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:07.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:07.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:07.707+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:07.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:07.708+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:07.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:07.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:07.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:07.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:08.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:08.402+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:08.404001+0000) 2024-02-19T20:07:08.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:08.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:08.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:08.613+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:08.614177+0000) 2024-02-19T20:07:08.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:08.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:08.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:08.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:08.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:08.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:08.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:08.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:09.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:08.877+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:08.878663+0000) 2024-02-19T20:07:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:08.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:08.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:08.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:08.989+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:09.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:09.405+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:09.406249+0000) 2024-02-19T20:07:09.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:09.614+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:09.615486+0000) 2024-02-19T20:07:09.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:09.855+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:09.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:09.879+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:09.880447+0000) 2024-02-19T20:07:09.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:09.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:09.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:09.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:09.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:09.766+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:09.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:09.767+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:09.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:09.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:09.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:09.972+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:10.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:10.405+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:10.406441+0000) 2024-02-19T20:07:10.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:10.614+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:10.615709+0000) 2024-02-19T20:07:10.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:10.879+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:10.880630+0000) 2024-02-19T20:07:10.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:10.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:10.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:10.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:10.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:10.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:10.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:10.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:10.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:10.791+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:11.237 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:10.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:11.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:10.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:11.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:10.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:11.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:10.984+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:11.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:11.241+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54127 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:11.239+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:11.240474+0000) 2024-02-19T20:07:11.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:11.238+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:11.240012+0000) 2024-02-19T20:07:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:11.239+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:11.240529+0000) 2024-02-19T20:07:11.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:11.405+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:11.406653+0000) 2024-02-19T20:07:11.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:11.844+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:11.881 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:11.614+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:11.615882+0000) 2024-02-19T20:07:11.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:11.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:11.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:11.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:11.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:11.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:11.882 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:11.794+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:12.209 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:11.879+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:11.880774+0000) 2024-02-19T20:07:12.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:12.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:12.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:12.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.005+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:12.407+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:12.408742+0000) 2024-02-19T20:07:12.883 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:12.854+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:12.884 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.617+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:12.618794+0000) 2024-02-19T20:07:12.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:12.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:12.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:12.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:12.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:12.753+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:12.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:12.753+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:13.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:12.882+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:12.883173+0000) 2024-02-19T20:07:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:12.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:13.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:13.409+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:13.410825+0000) 2024-02-19T20:07:13.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:13.619+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:13.620930+0000) 2024-02-19T20:07:13.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:13.826+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:13.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:13.884+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:13.885733+0000) 2024-02-19T20:07:13.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:13.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:13.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:13.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:13.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:13.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:13.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:13.748+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:14.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:14.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:14.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:14.020+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:14.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:14.621+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:14.622360+0000) 2024-02-19T20:07:14.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:14.411+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:14.412126+0000) 2024-02-19T20:07:14.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:14.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:14.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:14.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:14.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:14.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:14.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:14.699+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:14.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:14.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:14.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:14.887+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:14.888166+0000) 2024-02-19T20:07:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:15.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:15.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:15.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:15.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:15.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:15.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:15.621+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:15.622537+0000) 2024-02-19T20:07:15.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:15.411+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:15.412287+0000) 2024-02-19T20:07:15.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:15.776+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:15.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:15.887+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:15.888324+0000) 2024-02-19T20:07:15.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:15.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:15.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:15.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:15.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:15.684+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:15.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:15.685+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:16.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:16.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:16.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:16.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:16.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:16.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:16.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:16.026+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:16.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:16.248+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54166 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:16.245+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:16.246358+0000) 2024-02-19T20:07:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:16.246+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:16.247476+0000) 2024-02-19T20:07:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:16.246+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:16.247323+0000) 2024-02-19T20:07:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:16.411+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:16.412509+0000) 2024-02-19T20:07:16.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:16.817+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:16.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:16.621+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:16.622761+0000) 2024-02-19T20:07:16.889 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:16.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:16.889 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:16.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:16.890 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:16.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:43.008711+0000 (oldest deadline 2024-02-19T19:57:03.008711+0000) 2024-02-19T20:07:16.890 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:16.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:16.887+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:16.888472+0000) 2024-02-19T20:07:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:17.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:17.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:17.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:17.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:17.026+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:17.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:17.623+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:17.624874+0000) 2024-02-19T20:07:17.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:17.415+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:17.416083+0000) 2024-02-19T20:07:17.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:17.640+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:17.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:17.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:17.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:17.889+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:17.890693+0000) 2024-02-19T20:07:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:18.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:18.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:18.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:18.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:18.628 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:18.417+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:18.418095+0000) 2024-02-19T20:07:18.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:18.756+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:18.894 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:18.627+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:18.628436+0000) 2024-02-19T20:07:18.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:18.634+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:18.891+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:18.892923+0000) 2024-02-19T20:07:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:19.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:19.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:19.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T19:56:45.578918+0000 (oldest deadline 2024-02-19T19:57:05.578918+0000) 2024-02-19T20:07:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:19.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:19.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:19.419+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:19.420408+0000) 2024-02-19T20:07:19.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:19.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:19.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:19.628+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:19.629889+0000) 2024-02-19T20:07:19.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:19.645+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:20.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:19.893+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:19.894316+0000) 2024-02-19T20:07:20.172 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:20.001+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:20.419+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:20.420557+0000) 2024-02-19T20:07:20.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:20.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:20.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:20.753+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:20.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:20.629+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:20.630074+0000) 2024-02-19T20:07:21.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:20.893+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:20.894487+0000) 2024-02-19T20:07:21.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:20.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:21.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:21.254+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54207 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:21.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:21.252+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:21.253083+0000) 2024-02-19T20:07:21.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:21.252+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:21.253238+0000) 2024-02-19T20:07:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:21.253+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:21.254540+0000) 2024-02-19T20:07:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:21.419+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:21.420831+0000) 2024-02-19T20:07:21.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:21.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:21.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:21.705+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:21.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:21.629+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:21.630265+0000) 2024-02-19T20:07:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:21.893+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:21.894715+0000) 2024-02-19T20:07:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:21.974+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:22.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:22.422+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:22.423660+0000) 2024-02-19T20:07:22.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:22.753+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:22.897 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:22.631+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:22.632353+0000) 2024-02-19T20:07:22.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:22.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:22.895+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:22.896910+0000) 2024-02-19T20:07:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:23.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:23.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:23.425+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:23.426329+0000) 2024-02-19T20:07:23.899 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:23.758+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:23.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:23.633+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:23.634471+0000) 2024-02-19T20:07:23.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:23.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:23.898+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:23.899169+0000) 2024-02-19T20:07:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:24.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:24.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:24.426+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:24.427856+0000) 2024-02-19T20:07:24.899 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:24.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:24.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:24.655+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:24.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:24.635+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:24.636083+0000) 2024-02-19T20:07:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:25.007+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:24.899+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:24.900585+0000) 2024-02-19T20:07:25.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:25.427+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:25.428081+0000) 2024-02-19T20:07:25.899 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:25.635+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:25.636253+0000) 2024-02-19T20:07:25.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:25.676+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:25.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:25.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:26.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:25.899+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:25.900796+0000) 2024-02-19T20:07:26.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:25.966+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:26.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:26.259+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:26.260033+0000) 2024-02-19T20:07:26.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:26.262+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54245 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:26.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:26.259+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:26.260032+0000) 2024-02-19T20:07:26.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:26.258+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:26.259955+0000) 2024-02-19T20:07:26.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:26.427+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:26.428252+0000) 2024-02-19T20:07:26.899 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:26.669+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:26.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:26.635+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:26.636408+0000) 2024-02-19T20:07:26.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:26.680+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:26.899+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:26.900969+0000) 2024-02-19T20:07:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:27.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:27.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:27.429+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:27.430865+0000) 2024-02-19T20:07:27.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:27.651+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:27.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:27.637+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:27.638943+0000) 2024-02-19T20:07:27.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:27.727+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:27.902+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:27.903351+0000) 2024-02-19T20:07:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:27.992+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:28.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:28.432+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:28.433353+0000) 2024-02-19T20:07:28.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:28.640+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:28.641370+0000) 2024-02-19T20:07:28.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:28.774+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:28.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:28.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:28.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:28.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:28.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:28.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:28.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:28.678+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:28.904+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:28.905407+0000) 2024-02-19T20:07:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:29.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:29.643 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:29.433+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:29.434804+0000) 2024-02-19T20:07:29.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:29.642+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:29.643202+0000) 2024-02-19T20:07:29.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:29.782+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:29.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:29.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:29.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:29.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:29.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:29.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:29.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:29.683+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:30.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:29.905+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:29.906860+0000) 2024-02-19T20:07:30.172 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:29.990+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:30.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:30.642+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:30.643352+0000) 2024-02-19T20:07:30.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:30.433+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:30.435001+0000) 2024-02-19T20:07:30.958 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:30.748+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:30.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:30.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:30.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:30.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:30.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:30.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:30.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:30.688+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:30.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:30.906+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:30.907065+0000) 2024-02-19T20:07:31.264 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:30.986+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:31.267+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:31.268161+0000) 2024-02-19T20:07:31.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:31.269+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54285 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:31.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:31.266+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:31.267202+0000) 2024-02-19T20:07:31.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:31.267+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:31.268227+0000) 2024-02-19T20:07:31.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:31.434+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:31.435223+0000) 2024-02-19T20:07:31.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:31.642+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:31.643522+0000) 2024-02-19T20:07:31.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:31.717+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:31.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:31.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:31.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:31.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:31.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:31.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:31.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:31.681+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:31.906+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:31.907228+0000) 2024-02-19T20:07:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:31.994+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:32.646 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:32.436+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:32.437227+0000) 2024-02-19T20:07:32.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:32.644+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:32.645568+0000) 2024-02-19T20:07:32.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:32.765+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:32.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:32.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:32.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:32.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:32.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:32.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:32.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:32.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:32.908+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:32.909300+0000) 2024-02-19T20:07:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:32.961+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:33.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:33.438+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:33.439240+0000) 2024-02-19T20:07:33.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:33.646+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:33.647667+0000) 2024-02-19T20:07:33.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:33.783+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:33.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:33.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:33.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:33.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:33.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:33.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:33.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:33.710+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:33.980+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:33.910+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:33.911440+0000) 2024-02-19T20:07:34.649 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:34.439+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:34.440948+0000) 2024-02-19T20:07:34.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:34.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:34.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:34.648+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:34.649075+0000) 2024-02-19T20:07:34.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:34.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:34.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:34.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:34.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:34.749+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:34.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:34.750+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:34.966+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:34.911+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:34.912776+0000) 2024-02-19T20:07:35.649 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:35.440+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:35.441151+0000) 2024-02-19T20:07:35.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:35.783+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:35.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:35.648+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:35.649252+0000) 2024-02-19T20:07:35.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:35.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:35.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:35.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:35.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:35.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:35.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:35.704+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:36.272 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:35.911+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:35.912947+0000) 2024-02-19T20:07:36.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:35.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:36.273+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:36.274604+0000) 2024-02-19T20:07:36.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:36.275+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54323 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:36.273+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:36.274912+0000) 2024-02-19T20:07:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:36.274+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:36.275777+0000) 2024-02-19T20:07:36.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:36.440+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:36.441324+0000) 2024-02-19T20:07:36.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:36.648+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:36.649426+0000) 2024-02-19T20:07:36.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:36.827+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:36.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:36.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:36.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:36.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:36.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:36.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:36.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:36.724+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:36.912+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:36.913101+0000) 2024-02-19T20:07:37.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:36.925+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:37.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:37.442+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:37.443707+0000) 2024-02-19T20:07:37.916 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:37.650+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:37.651518+0000) 2024-02-19T20:07:37.917 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:37.845+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:37.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:37.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:37.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:37.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:37.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:37.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:37.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:37.677+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:37.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:37.914+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:37.915593+0000) 2024-02-19T20:07:38.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:38.445+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:38.446233+0000) 2024-02-19T20:07:38.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:38.652+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:38.653789+0000) 2024-02-19T20:07:38.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:38.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:38.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:38.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:38.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:38.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:38.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:38.801+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:38.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:38.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:38.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:38.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:38.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:38.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:38.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:38.692+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:38.916+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:38.917990+0000) 2024-02-19T20:07:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:38.937+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:39.655 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:39.446+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:39.447557+0000) 2024-02-19T20:07:39.919 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:39.654+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:39.655260+0000) 2024-02-19T20:07:39.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:39.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:39.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:39.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:39.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:39.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:39.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:39.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:39.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:39.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:39.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:39.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:39.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:39.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:39.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:39.681+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:40.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:39.918+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:39.919359+0000) 2024-02-19T20:07:40.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:39.946+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:40.713 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:40.654+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:40.655420+0000) 2024-02-19T20:07:40.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:40.446+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:40.447757+0000) 2024-02-19T20:07:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:40.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:40.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:40.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:40.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:41.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:40.848+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:40.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:40.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:41.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:40.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:40.713+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:40.918+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:40.919526+0000) 2024-02-19T20:07:41.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:41.282+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54363 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:41.388 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:41.280+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:41.281168+0000) 2024-02-19T20:07:41.388 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:41.279+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:41.280721+0000) 2024-02-19T20:07:41.388 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:41.280+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:41.281759+0000) 2024-02-19T20:07:41.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:41.446+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:41.447926+0000) 2024-02-19T20:07:41.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:41.654+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:41.655641+0000) 2024-02-19T20:07:41.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:41.915+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:41.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:41.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:41.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:41.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:41.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:41.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:41.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:41.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:41.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:41.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:41.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:41.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:41.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:41.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:41.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:41.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:42.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:41.918+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:41.919713+0000) 2024-02-19T20:07:42.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:42.449+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:42.450345+0000) 2024-02-19T20:07:42.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:42.658+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:42.659120+0000) 2024-02-19T20:07:42.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:42.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:42.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:42.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:42.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:42.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:42.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:42.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:42.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:42.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:42.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:42.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:42.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:42.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:42.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:42.697+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:42.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:42.922+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:42.923272+0000) 2024-02-19T20:07:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:42.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:42.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:42.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:42.937+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:43.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:43.451+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:43.452373+0000) 2024-02-19T20:07:43.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:43.660+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:43.661252+0000) 2024-02-19T20:07:43.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:43.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:43.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:43.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:43.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:43.876+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:43.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:43.877+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:43.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:43.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:43.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:43.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:43.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:43.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:43.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:43.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:43.924+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:43.925837+0000) 2024-02-19T20:07:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:43.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:43.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:43.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:43.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:44.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:44.453+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:44.454036+0000) 2024-02-19T20:07:44.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:44.661+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:44.662896+0000) 2024-02-19T20:07:44.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:44.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:44.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:44.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:44.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:44.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:44.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:44.887+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:44.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:44.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:44.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:44.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:44.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:44.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:44.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:44.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:44.926+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:44.927399+0000) 2024-02-19T20:07:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:44.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:44.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:44.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:44.981+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:45.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:45.453+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:45.454189+0000) 2024-02-19T20:07:45.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:45.662+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:45.663062+0000) 2024-02-19T20:07:45.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:45.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:45.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:45.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:45.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:45.761+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:45.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:45.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:45.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:45.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:45.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:45.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:45.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:45.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:45.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:45.839+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:46.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:45.926+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:45.927571+0000) 2024-02-19T20:07:46.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:45.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:46.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:45.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:46.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:45.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:46.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:45.977+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:46.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:46.289+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54401 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:46.286+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:46.287624+0000) 2024-02-19T20:07:46.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:46.286+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:46.287714+0000) 2024-02-19T20:07:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:46.287+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:46.288190+0000) 2024-02-19T20:07:46.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:46.453+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:46.454355+0000) 2024-02-19T20:07:46.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:46.662+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:46.663236+0000) 2024-02-19T20:07:46.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:46.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:46.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:46.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:46.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:46.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:46.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:46.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:46.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:46.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:46.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:46.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:46.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:46.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:46.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:46.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:46.926+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:46.927790+0000) 2024-02-19T20:07:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.023+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:47.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:47.455+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:47.456365+0000) 2024-02-19T20:07:47.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.664+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:47.665236+0000) 2024-02-19T20:07:47.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:47.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:47.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:47.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:47.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:47.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:47.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:47.777+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:47.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:47.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:47.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:47.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:47.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:47.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:47.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:47.803+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:47.928+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:47.929873+0000) 2024-02-19T20:07:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:47.982+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:48.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:48.458+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:48.459960+0000) 2024-02-19T20:07:48.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:48.667+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:48.668547+0000) 2024-02-19T20:07:48.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:48.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:48.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:48.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:48.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:48.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:48.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:48.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:48.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:48.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:48.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:48.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:48.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:48.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:48.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:48.833+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:49.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:48.932+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:48.933322+0000) 2024-02-19T20:07:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:48.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:48.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:48.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:48.962+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:49.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:49.460+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:49.461316+0000) 2024-02-19T20:07:49.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:49.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:49.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:49.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:49.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:49.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:49.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:49.732+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:49.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:49.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:49.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:49.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:49.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:49.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:49.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:49.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:49.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:49.669+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:49.670946+0000) 2024-02-19T20:07:50.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:49.934+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:49.935565+0000) 2024-02-19T20:07:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:49.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:49.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:49.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:50.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:49.967+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:50.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:50.461+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:50.461508+0000) 2024-02-19T20:07:50.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:50.671+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:50.671168+0000) 2024-02-19T20:07:50.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:50.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:50.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:50.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:50.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:50.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:50.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:50.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:50.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:50.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:50.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:50.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:50.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:50.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:50.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:50.756+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:50.935+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:50.935771+0000) 2024-02-19T20:07:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:51.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:51.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:51.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:51.012+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:51.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:51.296+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54441 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:51.625 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:51.293+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:51.294524+0000) 2024-02-19T20:07:51.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:51.294+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:51.295175+0000) 2024-02-19T20:07:51.626 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:51.294+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:51.295715+0000) 2024-02-19T20:07:51.626 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:51.460+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:51.461776+0000) 2024-02-19T20:07:51.902 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:51.670+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:51.671385+0000) 2024-02-19T20:07:51.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:51.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:51.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:51.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:51.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:51.719+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:51.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:51.720+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:51.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:51.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:51.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:51.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:51.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:51.810+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:51.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:51.811+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:51.935+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:51.936048+0000) 2024-02-19T20:07:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:52.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:52.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:52.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:52.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:52.675 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:52.463+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:52.464005+0000) 2024-02-19T20:07:52.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:52.673+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:52.674985+0000) 2024-02-19T20:07:52.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:52.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:52.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:52.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:52.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:52.680+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:52.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:52.681+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:52.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:52.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:52.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:52.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:52.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:52.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:52.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:52.782+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:52.937+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:52.938788+0000) 2024-02-19T20:07:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:53.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:53.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:53.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:53.080+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:53.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:53.465+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:53.466103+0000) 2024-02-19T20:07:53.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:53.676+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:53.677082+0000) 2024-02-19T20:07:53.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:53.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:53.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:53.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:53.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:53.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:53.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:53.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:53.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:53.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:53.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:53.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:53.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:53.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:53.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:53.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:53.940+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:53.941299+0000) 2024-02-19T20:07:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:54.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:54.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:54.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:54.110+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:54.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:54.466+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:54.467478+0000) 2024-02-19T20:07:54.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:54.677+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:54.678496+0000) 2024-02-19T20:07:54.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:54.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:54.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:54.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:54.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:54.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:54.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:54.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:54.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:54.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:54.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:54.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:54.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:54.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:54.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:54.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:55.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:54.942+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:54.943763+0000) 2024-02-19T20:07:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:55.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:55.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:55.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:55.131+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:55.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:55.466+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:55.467764+0000) 2024-02-19T20:07:55.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:55.677+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:55.678681+0000) 2024-02-19T20:07:55.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:55.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:55.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:55.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:55.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:55.717+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:55.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:55.718+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:55.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:55.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:55.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:55.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:55.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:55.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:55.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:55.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:56.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:55.943+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:55.944040+0000) 2024-02-19T20:07:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:56.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:56.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:56.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:56.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:56.625 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:07:56.303+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54479 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:07:56.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:56.300+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:56.301968+0000) 2024-02-19T20:07:56.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:56.301+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:56.302442+0000) 2024-02-19T20:07:56.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:56.301+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:56.302588+0000) 2024-02-19T20:07:56.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:56.466+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:56.467929+0000) 2024-02-19T20:07:56.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:56.677+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:56.678868+0000) 2024-02-19T20:07:56.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:56.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:56.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:56.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:56.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:56.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:56.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:56.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:56.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:56.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:56.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:56.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:56.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:56.732+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:56.904 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:56.733+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:57.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:56.943+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:56.944239+0000) 2024-02-19T20:07:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:57.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:57.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:57.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:57.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:57.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:57.469+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:57.470049+0000) 2024-02-19T20:07:57.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:57.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:57.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:57.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:57.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:57.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:57.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:57.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:57.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:57.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:57.948 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:57.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:57.948 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:57.747+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:57.948 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:57.747+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:57.948 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:57.681+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:57.682449+0000) 2024-02-19T20:07:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:57.945+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:57.946368+0000) 2024-02-19T20:07:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:58.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:58.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:58.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:58.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:58.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:58.471+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:58.472780+0000) 2024-02-19T20:07:58.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:58.684+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:58.685169+0000) 2024-02-19T20:07:58.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:58.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:58.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:58.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:58.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:58.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:58.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:58.783+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:58.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:58.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:58.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:58.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:58.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:58.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:58.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:58.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:59.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:58.947+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:58.948875+0000) 2024-02-19T20:07:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:59.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:59.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:59.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:07:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:59.097+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:07:59.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:59.473+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:59.474343+0000) 2024-02-19T20:07:59.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:07:59.685+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:59.686741+0000) 2024-02-19T20:07:59.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:59.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:59.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:59.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:59.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:59.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:07:59.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:59.789+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:07:59.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:59.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:59.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:59.714+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:07:59.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:07:59.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:07:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:07:59.949+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:07:59.950729+0000) 2024-02-19T20:08:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:00.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:00.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:00.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:00.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:00.145+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:00.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:00.473+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:00.474496+0000) 2024-02-19T20:08:00.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:00.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:00.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:00.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:00.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:00.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:00.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:00.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:00.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:00.685+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:00.686900+0000) 2024-02-19T20:08:00.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:00.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:00.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:00.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:00.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:00.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:00.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:00.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:00.949+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:00.950906+0000) 2024-02-19T20:08:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:01.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:01.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:01.107+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:01.108+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:01.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:01.308+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:01.309479+0000) 2024-02-19T20:08:01.627 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:01.310+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54515 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:01.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:01.308+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:01.309124+0000) 2024-02-19T20:08:01.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:01.308+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:01.309079+0000) 2024-02-19T20:08:01.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:01.473+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:01.474751+0000) 2024-02-19T20:08:01.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:01.686+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:01.687125+0000) 2024-02-19T20:08:01.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:01.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:01.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:01.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:01.757+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:01.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:01.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:01.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:01.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:01.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:01.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:01.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:01.693+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:02.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:01.950+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:01.951118+0000) 2024-02-19T20:08:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:02.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:02.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:02.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:02.064+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:02.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:02.475+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:02.476875+0000) 2024-02-19T20:08:02.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:02.688+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:02.689675+0000) 2024-02-19T20:08:02.954 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:02.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:02.954 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:02.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:02.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:02.720+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:02.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:02.720+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:02.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:02.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:02.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:02.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:02.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:02.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:02.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:02.792+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:02.952+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:02.953752+0000) 2024-02-19T20:08:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:03.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:03.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:03.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:03.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:03.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:03.478+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:03.479379+0000) 2024-02-19T20:08:03.956 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:03.692+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:03.693107+0000) 2024-02-19T20:08:03.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:03.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:03.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:03.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:03.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:03.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:03.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:03.773+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:03.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:03.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:03.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:03.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:03.958 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:03.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:03.958 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:03.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:04.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:03.955+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:03.956209+0000) 2024-02-19T20:08:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:04.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:04.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:04.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:04.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:04.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:04.479+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:04.480779+0000) 2024-02-19T20:08:04.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:04.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:04.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:04.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:04.958 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:04.693+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:04.694502+0000) 2024-02-19T20:08:04.959 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:04.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:04.959 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:04.680+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:04.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:04.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:04.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:04.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:04.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:04.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:04.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:04.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:04.957+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:04.958060+0000) 2024-02-19T20:08:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:05.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:05.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:05.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:05.147+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:05.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:05.479+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:05.480996+0000) 2024-02-19T20:08:05.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:05.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:05.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:05.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:05.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:05.646+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:05.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:05.647+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:05.958 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:05.693+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:05.694768+0000) 2024-02-19T20:08:05.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:05.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:05.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:05.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:05.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:05.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:05.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:05.737+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:05.957+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:05.958228+0000) 2024-02-19T20:08:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:06.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:06.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:06.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:06.102+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:06.627 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:06.317+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54553 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:06.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:06.315+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:06.316177+0000) 2024-02-19T20:08:06.628 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:06.315+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:06.316948+0000) 2024-02-19T20:08:06.628 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:06.315+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:06.316533+0000) 2024-02-19T20:08:06.628 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:06.480+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:06.481250+0000) 2024-02-19T20:08:06.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:06.694+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:06.695050+0000) 2024-02-19T20:08:06.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:06.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:06.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:06.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:06.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:06.764+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:06.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:06.765+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:06.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:06.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:06.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:06.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:06.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:06.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:06.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:06.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:06.957+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:06.958409+0000) 2024-02-19T20:08:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:07.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:07.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:07.072+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:07.074+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:07.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:07.482+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:07.483283+0000) 2024-02-19T20:08:07.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:07.696+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:07.697559+0000) 2024-02-19T20:08:07.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:07.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:07.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:07.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:07.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:07.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:07.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:07.745+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:07.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:07.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:07.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:07.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:07.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:07.710+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:07.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:07.710+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:07.959+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:07.960566+0000) 2024-02-19T20:08:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:08.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:08.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:08.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:08.088+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:08.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:08.484+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:08.485319+0000) 2024-02-19T20:08:08.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:08.699+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:08.700089+0000) 2024-02-19T20:08:08.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:08.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:08.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:08.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:08.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:08.727+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:08.905 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:08.728+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:08.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:08.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:08.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:08.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:08.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:08.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:08.906 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:08.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:09.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:08.961+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:08.962859+0000) 2024-02-19T20:08:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:09.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:09.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:09.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:09.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:09.072+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:09.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:09.485+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:09.486767+0000) 2024-02-19T20:08:09.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:09.700+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:09.701885+0000) 2024-02-19T20:08:09.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:09.775+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:09.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:09.775+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:09.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:09.775+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:09.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:09.775+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:09.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:09.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:09.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:09.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:09.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:09.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:09.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:09.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:10.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:09.963+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:09.964246+0000) 2024-02-19T20:08:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:10.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:10.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:10.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:10.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:10.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:10.087+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:10.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:10.485+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:10.486992+0000) 2024-02-19T20:08:10.962 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:10.701+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:10.702156+0000) 2024-02-19T20:08:10.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:10.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:10.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:10.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:10.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:10.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:10.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:10.793+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:10.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:10.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:10.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:10.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:10.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:10.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:10.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:10.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:10.963+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:10.964414+0000) 2024-02-19T20:08:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:11.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:11.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:11.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:11.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:11.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:11.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:11.088+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:11.627 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:11.324+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54593 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:11.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:11.323+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:11.324040+0000) 2024-02-19T20:08:11.628 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:11.321+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:11.322040+0000) 2024-02-19T20:08:11.628 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:11.320+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:11.321936+0000) 2024-02-19T20:08:11.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:11.486+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:11.487157+0000) 2024-02-19T20:08:11.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:11.701+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:11.702417+0000) 2024-02-19T20:08:11.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:11.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:11.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:11.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:11.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:11.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:11.905 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:11.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:11.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:11.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:11.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:11.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:11.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:11.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:11.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:11.764+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:12.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:11.963+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:11.964586+0000) 2024-02-19T20:08:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:12.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:12.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:12.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:12.124+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:12.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:12.488+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:12.489839+0000) 2024-02-19T20:08:12.963 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:12.704+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:12.705825+0000) 2024-02-19T20:08:12.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:12.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:12.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:12.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:12.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:12.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:12.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:12.757+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:12.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:12.763+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:12.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:12.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:12.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:12.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:12.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:12.764+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:13.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:12.965+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:12.966734+0000) 2024-02-19T20:08:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:13.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:13.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:13.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:13.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:13.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:13.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:13.492+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:13.493412+0000) 2024-02-19T20:08:13.963 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:13.708+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:13.709245+0000) 2024-02-19T20:08:13.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:13.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:13.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:13.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:13.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:13.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:13.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:13.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:13.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:13.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:13.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:13.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:13.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:13.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:13.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:13.763+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:14.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:13.967+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:13.968945+0000) 2024-02-19T20:08:14.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:14.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:14.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:14.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:14.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:14.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:14.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:14.224+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:14.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:14.710+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:14.711611+0000) 2024-02-19T20:08:14.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:14.493+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:14.494958+0000) 2024-02-19T20:08:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:14.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:14.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:14.781+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:15.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:14.781+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:14.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:14.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:14.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:14.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:15.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:14.969+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:14.970467+0000) 2024-02-19T20:08:15.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:15.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:15.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:15.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:15.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:15.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:15.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:15.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:15.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:15.494+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:15.495151+0000) 2024-02-19T20:08:15.963 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:15.710+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:15.711808+0000) 2024-02-19T20:08:15.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:15.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:15.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:15.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:15.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:15.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:15.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:15.782+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:15.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:15.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:15.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:15.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:15.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:15.788+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:15.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:15.789+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:16.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:15.969+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:15.970629+0000) 2024-02-19T20:08:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:16.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:16.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:16.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:16.200+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:16.628 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:16.331+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54631 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:16.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:16.330+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:16.331018+0000) 2024-02-19T20:08:16.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:16.328+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:16.329381+0000) 2024-02-19T20:08:16.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:16.494+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:16.495329+0000) 2024-02-19T20:08:16.629 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:16.328+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:16.329469+0000) 2024-02-19T20:08:16.905 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:16.710+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:16.712005+0000) 2024-02-19T20:08:16.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:16.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:16.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:16.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:16.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:16.777+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:16.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:16.777+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:16.906 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:16.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:16.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:16.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:16.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:16.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:16.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:16.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:17.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:16.969+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:16.970826+0000) 2024-02-19T20:08:17.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:17.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:17.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:17.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:17.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:17.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:17.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:17.169+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:17.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:17.496+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:17.497464+0000) 2024-02-19T20:08:17.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:17.713+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:17.714058+0000) 2024-02-19T20:08:18.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:17.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:17.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:17.778+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:18.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:17.778+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:17.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:17.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:17.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:18.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:17.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:18.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:17.971+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:17.972993+0000) 2024-02-19T20:08:18.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:18.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:18.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:18.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:18.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:18.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:18.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:18.217+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:18.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:18.499+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:18.500039+0000) 2024-02-19T20:08:18.963 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:18.716+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:18.717571+0000) 2024-02-19T20:08:18.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:18.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:18.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:18.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:18.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:18.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:18.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:18.796+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:18.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:18.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:18.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:18.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:18.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:18.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:18.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:18.808+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:19.236 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:18.974+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:18.975185+0000) 2024-02-19T20:08:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:19.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:19.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:19.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:19.235+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:19.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:19.718+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:19.719936+0000) 2024-02-19T20:08:19.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:19.500+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:19.501467+0000) 2024-02-19T20:08:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:19.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:19.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:19.829+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:19.829+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:19.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:19.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:19.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:19.777+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:20.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:19.975+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:19.976577+0000) 2024-02-19T20:08:20.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:20.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:20.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:20.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:20.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:20.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:20.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:20.208+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:20.688 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:20.500+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:20.501640+0000) 2024-02-19T20:08:20.963 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:20.719+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:20.720126+0000) 2024-02-19T20:08:20.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:20.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:20.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:20.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:20.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:20.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:20.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:20.813+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:20.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:20.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:20.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:20.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:20.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:20.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:20.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:20.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:21.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:20.975+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:20.976755+0000) 2024-02-19T20:08:21.502 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:21.338+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54671 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:21.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:21.336+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:21.337548+0000) 2024-02-19T20:08:21.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:21.335+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:21.336272+0000) 2024-02-19T20:08:21.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:21.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:21.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:21.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:21.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:21.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:21.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:21.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:21.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:21.335+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:21.336864+0000) 2024-02-19T20:08:21.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:21.719+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:21.720275+0000) 2024-02-19T20:08:21.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:21.500+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:21.501847+0000) 2024-02-19T20:08:21.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:21.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:21.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:21.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:21.780+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:22.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:21.781+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:22.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:21.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:21.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:21.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:21.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:22.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:21.975+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:21.976926+0000) 2024-02-19T20:08:22.414 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:22.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:22.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:22.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:22.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:22.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:22.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:22.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:22.688 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:22.503+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:22.504796+0000) 2024-02-19T20:08:22.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:22.721+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:22.722283+0000) 2024-02-19T20:08:22.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:22.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:22.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:22.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:22.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:22.798+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:22.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:22.799+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:22.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:22.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:22.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:22.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:22.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:22.737+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:22.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:22.737+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:23.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:22.978+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:22.979319+0000) 2024-02-19T20:08:23.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:23.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:23.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:23.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:23.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:23.226+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:23.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:23.227+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:23.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:23.723+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:23.724294+0000) 2024-02-19T20:08:23.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:23.506+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:23.507011+0000) 2024-02-19T20:08:23.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:23.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:23.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:23.701+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:23.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:23.702+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:24.109 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:23.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:24.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:23.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:24.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:23.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:24.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:23.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:24.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:23.980+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:23.981336+0000) 2024-02-19T20:08:24.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:24.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:24.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:24.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:24.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:24.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:24.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:24.184+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:24.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:24.508+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:24.509528+0000) 2024-02-19T20:08:24.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:24.724+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:24.725722+0000) 2024-02-19T20:08:24.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:24.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:24.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:24.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:24.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:24.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:24.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:24.804+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:24.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:24.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:24.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:24.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:24.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:24.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:24.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:24.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:24.981+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:24.982741+0000) 2024-02-19T20:08:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:25.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:25.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:25.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:25.168+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:25.688 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:25.508+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:25.509731+0000) 2024-02-19T20:08:25.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:25.724+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:25.725931+0000) 2024-02-19T20:08:25.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:25.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:25.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:25.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:25.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:25.757+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:25.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:25.758+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:25.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:25.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:25.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:25.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:25.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:25.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:25.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:25.848+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:26.216 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:25.981+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:25.982927+0000) 2024-02-19T20:08:26.510 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:26.345+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54709 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:26.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:26.343+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:26.344453+0000) 2024-02-19T20:08:26.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:26.342+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:26.343434+0000) 2024-02-19T20:08:26.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:26.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:26.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:26.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:26.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:26.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:26.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:26.216+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:26.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:26.343+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:26.344259+0000) 2024-02-19T20:08:26.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:26.725+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:26.726118+0000) 2024-02-19T20:08:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:26.508+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:26.509893+0000) 2024-02-19T20:08:26.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:26.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:26.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:26.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:26.742+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:27.211 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:26.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:27.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:26.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:27.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:26.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:27.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:26.898+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:27.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:26.982+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:26.983083+0000) 2024-02-19T20:08:27.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:27.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:27.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:27.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:27.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:27.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:27.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:27.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:27.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:27.727+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:27.728768+0000) 2024-02-19T20:08:27.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:27.511+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:27.512004+0000) 2024-02-19T20:08:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:27.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:27.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:27.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:27.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:27.695+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:28.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:28.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:28.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:28.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:28.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:28.214+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:28.217 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:27.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:28.217 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:27.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:28.217 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:27.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:28.217 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:27.942+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:28.217 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:27.984+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:27.985713+0000) 2024-02-19T20:08:28.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:28.215+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:28.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:28.729+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:28.730833+0000) 2024-02-19T20:08:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:28.513+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:28.514122+0000) 2024-02-19T20:08:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:28.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:28.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:28.661+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:28.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:29.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:28.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:29.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:28.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:29.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:28.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:29.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:28.980+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:29.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:28.987+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:28.988154+0000) 2024-02-19T20:08:29.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:29.201+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:29.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:29.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:29.202+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:29.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:29.732+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:29.733237+0000) 2024-02-19T20:08:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:29.514+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:29.515961+0000) 2024-02-19T20:08:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:29.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:29.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:29.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:29.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:29.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:30.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:29.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:30.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:29.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:30.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:29.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:30.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:29.942+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:30.177 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:29.988+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:29.989957+0000) 2024-02-19T20:08:30.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:30.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:30.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:30.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:30.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:30.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:30.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:30.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:30.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:30.732+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:30.733391+0000) 2024-02-19T20:08:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:30.515+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:30.516138+0000) 2024-02-19T20:08:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:30.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:30.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:30.698+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:30.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:30.699+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:30.899+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:30.899+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:30.899+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:30.900+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:30.989+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:30.990145+0000) 2024-02-19T20:08:31.629 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:31.352+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54749 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:31.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:31.350+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:31.351593+0000) 2024-02-19T20:08:31.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:31.349+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:31.350405+0000) 2024-02-19T20:08:31.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:31.515+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:31.516379+0000) 2024-02-19T20:08:31.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:31.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:31.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:31.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:31.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:31.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:31.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:31.293+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:31.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:31.350+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:31.351458+0000) 2024-02-19T20:08:31.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:31.732+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:31.733570+0000) 2024-02-19T20:08:31.907 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:31.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:31.907 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:31.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:31.907 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:31.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:31.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:31.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:31.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:31.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:31.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:31.943+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:31.989+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:31.990302+0000) 2024-02-19T20:08:32.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:32.517+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:32.518469+0000) 2024-02-19T20:08:32.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:32.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:32.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:32.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:32.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:32.313+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:32.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:32.313+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:32.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:32.735+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:32.736926+0000) 2024-02-19T20:08:32.907 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:32.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:32.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:32.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:32.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:32.705+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:32.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:32.705+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:32.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:32.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:32.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:32.963+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:32.991+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:32.992645+0000) 2024-02-19T20:08:33.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:33.519+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:33.520600+0000) 2024-02-19T20:08:33.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:33.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:33.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:33.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:33.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:33.360+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:33.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:33.361+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:33.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:33.738+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:33.739054+0000) 2024-02-19T20:08:33.907 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:33.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:33.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:33.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:33.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:33.676+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:33.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:33.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:33.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:33.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:33.957+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:33.958+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:33.993+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:33.994796+0000) 2024-02-19T20:08:34.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:34.521+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:34.522061+0000) 2024-02-19T20:08:34.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:34.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:34.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:34.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:34.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:34.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:34.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:34.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:34.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:34.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:34.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:34.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:34.686 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:34.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:34.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:34.683+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:34.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:34.740+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:34.741456+0000) 2024-02-19T20:08:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:34.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:34.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:34.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:34.984+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:35.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:34.995+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:34.996677+0000) 2024-02-19T20:08:35.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:35.521+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:35.522258+0000) 2024-02-19T20:08:35.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:35.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:35.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:35.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:35.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:35.634+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:35.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:35.634+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:35.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:35.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:35.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:35.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:35.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:35.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:35.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:35.418+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:35.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:35.740+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:35.741646+0000) 2024-02-19T20:08:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:35.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:35.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:35.938+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:35.939+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:35.995+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:35.996866+0000) 2024-02-19T20:08:36.630 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:36.360+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54788 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:36.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:36.357+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:36.358294+0000) 2024-02-19T20:08:36.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:36.356+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:36.357749+0000) 2024-02-19T20:08:36.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:36.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:36.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:36.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:36.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:36.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:36.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:36.418+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:36.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:36.357+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:36.358503+0000) 2024-02-19T20:08:36.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:36.521+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:36.522450+0000) 2024-02-19T20:08:36.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:36.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:36.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:36.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:36.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:36.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:36.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:36.601+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:36.907 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:36.740+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:36.741839+0000) 2024-02-19T20:08:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:36.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:36.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:36.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:36.928+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:37.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:36.996+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:36.997035+0000) 2024-02-19T20:08:37.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:37.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:37.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:37.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:37.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:37.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:37.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:37.390+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:37.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:37.523+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:37.524643+0000) 2024-02-19T20:08:37.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:37.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:37.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:37.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:37.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:37.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:37.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:37.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:37.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:37.744+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:37.744342+0000) 2024-02-19T20:08:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:37.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:37.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:37.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:37.972+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:38.000+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:38.000438+0000) 2024-02-19T20:08:38.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:38.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:38.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:38.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:38.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:38.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:38.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:38.374+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:38.631 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:38.526+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:38.526706+0000) 2024-02-19T20:08:38.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:38.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:38.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:38.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:38.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:38.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:38.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:38.539+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:38.908 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:38.745+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:38.746437+0000) 2024-02-19T20:08:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:38.973+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:38.973+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:38.973+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:38.974+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:39.002+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:39.003096+0000) 2024-02-19T20:08:39.631 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:39.527+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:39.528228+0000) 2024-02-19T20:08:39.631 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:39.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:39.631 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:39.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:39.631 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:39.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:39.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:39.529+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:39.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:39.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:39.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:39.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:39.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:39.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:39.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:39.338+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:39.908 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:39.747+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:39.748061+0000) 2024-02-19T20:08:40.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:39.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:40.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:39.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:40.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:39.971+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:40.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:39.972+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:40.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:40.003+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:40.004984+0000) 2024-02-19T20:08:40.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:40.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:40.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:40.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:40.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:40.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:40.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:40.388+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:40.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:40.747+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:40.748233+0000) 2024-02-19T20:08:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:40.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:40.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:40.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:40.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:40.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:40.527+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:40.528480+0000) 2024-02-19T20:08:41.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:40.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:41.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:40.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:41.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:40.972+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:41.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:40.973+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:41.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:41.004+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:41.005158+0000) 2024-02-19T20:08:41.631 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:41.367+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54827 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:41.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:41.364+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:41.365801+0000) 2024-02-19T20:08:41.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:41.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:41.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:41.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:41.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:41.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:41.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:41.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:41.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:41.364+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:41.365608+0000) 2024-02-19T20:08:41.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:41.364+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:41.365942+0000) 2024-02-19T20:08:41.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:41.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:41.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:41.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:41.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:41.523+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:41.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:41.523+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:41.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:41.527+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:41.528725+0000) 2024-02-19T20:08:41.908 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:41.747+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:41.748422+0000) 2024-02-19T20:08:42.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.004+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:42.005327+0000) 2024-02-19T20:08:42.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.006+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:42.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.006+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.006+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:42.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.007+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:42.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:42.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:42.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:42.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:42.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:42.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:42.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:42.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:42.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:42.530+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:42.531265+0000) 2024-02-19T20:08:42.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:42.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:42.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:42.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:42.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:42.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:42.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:42.542+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:42.966 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:42.749+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:42.750632+0000) 2024-02-19T20:08:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:42.977+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:43.006+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:43.007944+0000) 2024-02-19T20:08:43.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:43.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:43.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:43.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:43.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:43.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:43.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:43.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:43.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:43.532+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:43.533908+0000) 2024-02-19T20:08:43.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:43.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:43.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:43.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:43.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:43.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:43.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:43.557+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:43.908 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:43.751+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:43.752762+0000) 2024-02-19T20:08:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:44.010+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:44.011616+0000) 2024-02-19T20:08:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:44.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:44.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:44.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:44.022+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:44.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:44.023+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:44.631 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:44.534+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:44.535844+0000) 2024-02-19T20:08:44.631 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:44.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:44.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:44.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:44.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:44.601+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:44.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:44.602+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:44.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:44.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:44.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:44.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:44.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:44.352+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:44.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:44.353+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:44.908 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:44.754+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:44.755271+0000) 2024-02-19T20:08:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.012+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:45.013243+0000) 2024-02-19T20:08:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.044+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:45.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:45.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:45.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:45.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:45.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:45.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:45.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:45.384+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:45.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:45.535+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:45.536065+0000) 2024-02-19T20:08:45.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:45.593+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:45.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:45.593+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:45.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:45.593+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:45.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:45.594+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:45.966 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:45.754+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:45.755450+0000) 2024-02-19T20:08:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:45.998+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:46.012+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:46.013509+0000) 2024-02-19T20:08:46.631 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:46.374+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54867 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:46.632 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:46.371+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:46.372206+0000) 2024-02-19T20:08:46.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:46.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:46.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:46.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:46.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:46.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:46.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:46.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:46.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:46.372+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:46.373059+0000) 2024-02-19T20:08:46.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:46.372+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:46.373497+0000) 2024-02-19T20:08:46.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:46.535+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:46.536238+0000) 2024-02-19T20:08:46.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:46.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:46.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:46.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:46.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:46.592+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:46.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:46.592+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:46.908 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:46.754+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:46.755621+0000) 2024-02-19T20:08:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.006+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:47.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.012+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:47.013763+0000) 2024-02-19T20:08:47.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:47.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:47.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:47.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:47.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:47.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:47.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:47.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:47.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:47.537+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:47.538263+0000) 2024-02-19T20:08:47.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:47.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:47.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:47.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:47.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:47.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:47.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:47.547+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:47.966 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:47.756+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:47.757726+0000) 2024-02-19T20:08:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:47.987+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:48.014+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:48.015692+0000) 2024-02-19T20:08:48.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:48.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:48.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:48.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:48.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:48.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:48.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:48.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:48.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:48.528+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:48.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:48.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:48.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:48.529+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:48.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:48.529+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:48.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:48.539+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:48.540232+0000) 2024-02-19T20:08:48.909 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:48.760+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:48.761243+0000) 2024-02-19T20:08:49.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:48.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:49.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:48.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:49.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:48.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:49.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:48.965+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:49.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:49.016+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:49.017745+0000) 2024-02-19T20:08:49.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:49.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:49.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:49.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:49.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:49.503+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:49.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:49.504+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:49.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:49.540+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:49.541727+0000) 2024-02-19T20:08:49.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:49.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:49.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:49.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:49.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:49.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:49.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:49.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:49.909 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:49.762+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:49.763764+0000) 2024-02-19T20:08:50.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:50.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:50.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:50.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.001+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:50.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.018+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:50.019239+0000) 2024-02-19T20:08:50.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:50.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:50.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:50.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:50.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:50.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:50.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:50.377+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:50.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:50.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:50.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:50.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:50.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:50.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:50.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:50.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:50.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:50.540+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:50.541910+0000) 2024-02-19T20:08:50.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:50.762+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:50.763946+0000) 2024-02-19T20:08:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.997+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.997+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:50.997+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:51.018+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:51.019420+0000) 2024-02-19T20:08:51.632 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:51.378+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:51.379695+0000) 2024-02-19T20:08:51.632 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:51.381+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54906 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:51.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:51.377+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:51.378953+0000) 2024-02-19T20:08:51.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:51.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:51.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:51.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:51.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:51.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:51.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:51.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:51.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:51.379+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:51.380284+0000) 2024-02-19T20:08:51.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:51.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:51.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:51.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:51.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:51.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:51.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:51.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:51.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:51.541+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:51.542082+0000) 2024-02-19T20:08:51.909 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:51.763+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:51.764140+0000) 2024-02-19T20:08:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:51.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:51.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:51.962+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:51.963+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:52.018+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:52.019622+0000) 2024-02-19T20:08:52.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:52.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:52.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:52.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:52.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:52.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:52.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:52.440+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:52.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:52.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:52.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:52.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:52.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:52.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:52.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:52.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:52.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:52.543+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:52.544126+0000) 2024-02-19T20:08:52.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:52.765+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:52.766143+0000) 2024-02-19T20:08:52.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:52.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:52.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:52.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:52.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:52.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:52.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:52.920+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:53.021+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:53.022049+0000) 2024-02-19T20:08:53.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:53.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:53.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:53.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:53.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:53.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:53.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:53.490+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:53.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:53.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:53.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:53.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:53.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:53.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:53.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:53.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:53.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:53.545+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:53.546157+0000) 2024-02-19T20:08:53.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:53.767+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:53.768578+0000) 2024-02-19T20:08:53.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:53.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:53.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:53.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:53.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:53.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:53.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:53.919+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:54.023+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:54.024065+0000) 2024-02-19T20:08:54.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:54.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:54.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:54.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:54.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:54.465+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:54.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:54.465+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:54.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:54.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:54.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:54.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:54.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:54.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:54.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:54.489+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:54.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:54.546+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:54.547732+0000) 2024-02-19T20:08:54.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:54.770+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:54.771230+0000) 2024-02-19T20:08:54.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:54.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:54.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:54.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:54.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:54.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:54.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:54.874+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:55.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:55.024+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:55.025498+0000) 2024-02-19T20:08:55.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:55.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:55.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:55.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:55.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:55.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:55.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:55.474+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:55.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:55.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:55.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:55.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:55.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:55.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:55.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:55.472+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:55.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:55.546+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:55.547926+0000) 2024-02-19T20:08:55.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:55.770+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:55.771404+0000) 2024-02-19T20:08:55.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:55.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:55.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:55.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:55.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:55.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:55.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:55.863+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:56.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:56.024+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:56.025696+0000) 2024-02-19T20:08:56.692 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:08:56.387+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54945 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:08:56.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:56.385+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:56.386803+0000) 2024-02-19T20:08:56.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:56.385+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:56.386958+0000) 2024-02-19T20:08:56.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:56.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:56.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:56.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:56.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:56.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:56.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:56.444+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:56.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:56.385+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:56.386325+0000) 2024-02-19T20:08:56.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:56.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:56.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:56.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:56.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:56.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:56.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:56.442+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:56.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:56.547+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:56.548129+0000) 2024-02-19T20:08:56.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:56.770+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:56.771592+0000) 2024-02-19T20:08:56.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:56.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:56.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:56.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:56.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:56.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:56.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:56.901+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:57.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:57.024+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:57.025897+0000) 2024-02-19T20:08:57.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:57.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:57.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:57.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:57.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:57.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:57.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:57.441+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:57.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:57.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:57.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:57.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:57.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:57.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:57.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:57.467+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:57.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:57.549+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:57.550229+0000) 2024-02-19T20:08:57.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:57.773+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:57.775005+0000) 2024-02-19T20:08:57.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:57.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:57.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:57.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:57.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:57.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:57.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:57.899+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:58.028+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:58.029428+0000) 2024-02-19T20:08:58.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:58.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:58.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:58.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:58.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:58.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:58.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:58.459+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:58.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:58.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:58.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:58.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:58.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:58.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:58.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:58.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:58.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:58.551+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:58.552179+0000) 2024-02-19T20:08:58.967 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:58.777+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:58.778533+0000) 2024-02-19T20:08:58.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:58.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:58.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:58.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:58.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:58.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:58.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:58.896+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:59.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:59.032+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:59.033016+0000) 2024-02-19T20:08:59.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:59.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:59.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:59.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:59.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:59.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:08:59.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:59.482+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:08:59.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:59.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:59.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:59.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:59.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:59.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:08:59.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:59.383+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:08:59.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:08:59.552+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:59.553789+0000) 2024-02-19T20:08:59.968 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:08:59.780+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:08:59.781056+0000) 2024-02-19T20:08:59.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:59.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:59.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:59.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:59.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:59.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:08:59.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:08:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:08:59.887+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:00.033+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:00.034922+0000) 2024-02-19T20:09:00.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:00.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:00.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:00.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:00.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:00.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:00.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:00.459+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:00.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:00.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:00.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:00.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:00.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:00.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:00.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:00.339+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:00.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:00.552+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:00.553982+0000) 2024-02-19T20:09:00.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:00.780+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:00.781246+0000) 2024-02-19T20:09:00.910 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:00.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:00.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:00.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:00.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:00.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:00.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:00.840+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:01.034+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:01.035064+0000) 2024-02-19T20:09:01.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:01.391+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:01.392372+0000) 2024-02-19T20:09:01.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:01.394+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 54984 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:01.393+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:01.394099+0000) 2024-02-19T20:09:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:01.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:01.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:01.439+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:01.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:01.440+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:01.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:01.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:01.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:01.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:01.291+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:01.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:01.391+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:01.392366+0000) 2024-02-19T20:09:01.820 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:01.780+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:01.781425+0000) 2024-02-19T20:09:01.820 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:01.553+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:01.554153+0000) 2024-02-19T20:09:02.113 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:01.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:02.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:01.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:02.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:01.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:02.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:01.819+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:02.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:02.034+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:02.035252+0000) 2024-02-19T20:09:02.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:02.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:02.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:02.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:02.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:02.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:02.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:02.253+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:02.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:02.555+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:02.556198+0000) 2024-02-19T20:09:02.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:02.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:02.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:02.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:02.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:02.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:02.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:02.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:02.968 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:02.782+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:02.783503+0000) 2024-02-19T20:09:02.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:02.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:02.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:02.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:02.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:02.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:02.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:02.842+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:03.037+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:03.038799+0000) 2024-02-19T20:09:03.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:03.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:03.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:03.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:03.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:03.443+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:03.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:03.443+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:03.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:03.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:03.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:03.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:03.291+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:03.824 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:03.784+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:03.785666+0000) 2024-02-19T20:09:03.825 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:03.557+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:03.558184+0000) 2024-02-19T20:09:04.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:03.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:04.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:03.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:04.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:03.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:04.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:03.823+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:04.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:04.039+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:04.040936+0000) 2024-02-19T20:09:04.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:04.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:04.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:04.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:04.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:04.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:04.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:04.288+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:04.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:04.559+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:04.560065+0000) 2024-02-19T20:09:04.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:04.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:04.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:04.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:04.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:04.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:04.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:04.432+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:04.968 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:04.787+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:04.788306+0000) 2024-02-19T20:09:04.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:04.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:04.968 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:04.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:04.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:04.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:04.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:04.874+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:05.041+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:05.042441+0000) 2024-02-19T20:09:05.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:05.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:05.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:05.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:05.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:05.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:05.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:05.455+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:05.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:05.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:05.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:05.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:05.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:05.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:05.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:05.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:05.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:05.559+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:05.560236+0000) 2024-02-19T20:09:05.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:05.787+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:05.788498+0000) 2024-02-19T20:09:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:05.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:05.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:05.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:06.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:05.886+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:06.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:06.041+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:06.042654+0000) 2024-02-19T20:09:06.634 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:06.401+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55023 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:06.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:06.399+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:06.400595+0000) 2024-02-19T20:09:06.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:06.398+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:06.399940+0000) 2024-02-19T20:09:06.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:06.467+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:06.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:06.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:06.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:06.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:06.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:06.468+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:06.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:06.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:06.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:06.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:06.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:06.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:06.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:06.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:06.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:06.399+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:06.400015+0000) 2024-02-19T20:09:06.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:06.559+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:06.560429+0000) 2024-02-19T20:09:06.905 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:06.787+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:06.788685+0000) 2024-02-19T20:09:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:06.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:06.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:06.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:06.904+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:07.041+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:07.042821+0000) 2024-02-19T20:09:07.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:07.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:07.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:07.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:07.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:07.472+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:07.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:07.473+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:07.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:07.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:07.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:07.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:07.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:07.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:07.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:07.306+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:07.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:07.789+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:07.790799+0000) 2024-02-19T20:09:07.911 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:07.561+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:07.562427+0000) 2024-02-19T20:09:08.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:07.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:08.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:07.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:08.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:07.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:08.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:07.932+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:08.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:08.043+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:08.044804+0000) 2024-02-19T20:09:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:08.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:08.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:08.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:08.278+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:08.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:08.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:08.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:08.489+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:08.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:08.490+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:08.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:08.791+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:08.792986+0000) 2024-02-19T20:09:08.911 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:08.563+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:08.564972+0000) 2024-02-19T20:09:09.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:08.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:09.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:08.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:09.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:08.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:09.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:08.920+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:09.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:09.045+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:09.046816+0000) 2024-02-19T20:09:09.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:09.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:09.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:09.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:09.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:09.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:09.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:09.441+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:09.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:09.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:09.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:09.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:09.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:09.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:09.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:09.258+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:09.902 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:09.793+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:09.794925+0000) 2024-02-19T20:09:09.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:09.566+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:09.567733+0000) 2024-02-19T20:09:10.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:09.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:10.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:09.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:10.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:09.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:10.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:09.902+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:10.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:10.047+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:10.048300+0000) 2024-02-19T20:09:10.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:10.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:10.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:10.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:10.445 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:10.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:10.445 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:10.251+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:10.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:10.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:10.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:10.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:10.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:10.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:10.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:10.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:10.566+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:10.567939+0000) 2024-02-19T20:09:11.048 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:10.794+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:10.795120+0000) 2024-02-19T20:09:11.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:10.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:11.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:10.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:11.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:10.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:11.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:10.910+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:11.049 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:11.047+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:11.048466+0000) 2024-02-19T20:09:11.403 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:11.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:11.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:11.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:11.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:11.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:11.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:11.225+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:11.694 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:11.408+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55062 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:11.694 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:11.406+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:11.407154+0000) 2024-02-19T20:09:11.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:11.405+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:11.406774+0000) 2024-02-19T20:09:11.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:11.567+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:11.568124+0000) 2024-02-19T20:09:11.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:11.406+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:11.407093+0000) 2024-02-19T20:09:11.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:11.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:11.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:11.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:11.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:11.435+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:11.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:11.435+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:11.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:11.794+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:11.795311+0000) 2024-02-19T20:09:12.266 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:11.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:12.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:11.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:12.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:11.943+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:12.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:11.944+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:12.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:12.047+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:12.048630+0000) 2024-02-19T20:09:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:12.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:12.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:12.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:12.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:12.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:12.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:12.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:12.450+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:12.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:12.451+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:12.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:12.796+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:12.797322+0000) 2024-02-19T20:09:12.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:12.569+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:12.570587+0000) 2024-02-19T20:09:13.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:12.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:13.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:12.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:13.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:12.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:13.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:12.965+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:13.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:13.049+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:13.050640+0000) 2024-02-19T20:09:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:13.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:13.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:13.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:13.413+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:13.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:13.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:13.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:13.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:13.277+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:13.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:13.798+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:13.799452+0000) 2024-02-19T20:09:13.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:13.571+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:13.572579+0000) 2024-02-19T20:09:14.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:14.011+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:14.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:14.011+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:14.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:14.011+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:14.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:14.012+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:14.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:14.051+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:14.052635+0000) 2024-02-19T20:09:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:14.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:14.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:14.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:14.252+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:14.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:14.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:14.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:14.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:14.395+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:14.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:14.799+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:14.800979+0000) 2024-02-19T20:09:14.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:14.573+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:14.574100+0000) 2024-02-19T20:09:15.248 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:15.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:15.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.002+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:15.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.003+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:15.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.053+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:15.054166+0000) 2024-02-19T20:09:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:15.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:15.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:15.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:15.247+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:15.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:15.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:15.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:15.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:15.347+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:15.911 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:15.573+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:15.574259+0000) 2024-02-19T20:09:15.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:15.800+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:15.801165+0000) 2024-02-19T20:09:16.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:16.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:16.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:16.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:15.987+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:16.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:16.053+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:16.054321+0000) 2024-02-19T20:09:16.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:16.415+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55101 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:16.412+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:16.413484+0000) 2024-02-19T20:09:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:16.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:16.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:16.335+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:16.336+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:16.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:16.412+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:16.413443+0000) 2024-02-19T20:09:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:16.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:16.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:16.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:16.284+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:16.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:16.412+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:16.413468+0000) 2024-02-19T20:09:16.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:16.800+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:16.801365+0000) 2024-02-19T20:09:16.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:16.573+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:16.574419+0000) 2024-02-19T20:09:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:16.980+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:16.980+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:17.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:16.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:17.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:16.981+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:17.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:17.053+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:17.054475+0000) 2024-02-19T20:09:17.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:17.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:17.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:17.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:17.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:17.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:17.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:17.354+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:17.579 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:17.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:17.579 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:17.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:17.579 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:17.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:17.579 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:17.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:17.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:17.802+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:17.803995+0000) 2024-02-19T20:09:17.912 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:17.576+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:17.577767+0000) 2024-02-19T20:09:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:18.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:18.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:18.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:18.020+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:18.056+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:18.057050+0000) 2024-02-19T20:09:18.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:18.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:18.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:18.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:18.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:18.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:18.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:18.333+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:18.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:18.578+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:18.579826+0000) 2024-02-19T20:09:18.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:18.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:18.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:18.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:18.637 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:18.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:18.637 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:18.386+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:18.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:18.805+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:18.806396+0000) 2024-02-19T20:09:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:19.058+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:19.059070+0000) 2024-02-19T20:09:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:19.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:19.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:19.063+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:19.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:19.064+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:19.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:19.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:19.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:19.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:19.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:19.411+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:19.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:19.411+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:19.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:19.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:19.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:19.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:19.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:19.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:19.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:19.356+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:19.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:19.580+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:19.581521+0000) 2024-02-19T20:09:19.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:19.807+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:19.808351+0000) 2024-02-19T20:09:20.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:20.060+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:20.061728+0000) 2024-02-19T20:09:20.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:20.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:20.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:20.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:20.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:20.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:20.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:20.066+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:20.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:20.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:20.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:20.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:20.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:20.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:20.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:20.350+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:20.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:20.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:20.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:20.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:20.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:20.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:20.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:20.366+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:20.580+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:20.581711+0000) 2024-02-19T20:09:21.062 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:20.807+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:20.808529+0000) 2024-02-19T20:09:21.062 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:21.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:21.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:21.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:21.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:21.029+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:21.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:21.030+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:21.360 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:21.060+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:21.061946+0000) 2024-02-19T20:09:21.635 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:21.422+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55140 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:21.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:21.420+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:21.421439+0000) 2024-02-19T20:09:21.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:21.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:21.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:21.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:21.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:21.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:21.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:21.360+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:21.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:21.419+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:21.420215+0000) 2024-02-19T20:09:21.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:21.580+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:21.581899+0000) 2024-02-19T20:09:21.637 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:21.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:21.637 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:21.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:21.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:21.414+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:21.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:21.415+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:21.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:21.420+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:21.421167+0000) 2024-02-19T20:09:21.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:21.807+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:21.808723+0000) 2024-02-19T20:09:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:22.061+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:22.062134+0000) 2024-02-19T20:09:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:22.067+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:22.067+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:22.067+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:22.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:22.068+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:22.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:22.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:22.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:22.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:22.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:22.423+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:22.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:22.424+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:22.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:22.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:22.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:22.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:22.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:22.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:22.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:22.321+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:22.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:22.811+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:22.812233+0000) 2024-02-19T20:09:22.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:22.582+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:22.584009+0000) 2024-02-19T20:09:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:23.064+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:23.065510+0000) 2024-02-19T20:09:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:23.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:23.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:23.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:23.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:23.091+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:23.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:23.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:23.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:23.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:23.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:23.427+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:23.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:23.428+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:23.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:23.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:23.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:23.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:23.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:23.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:23.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:23.350+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:23.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:23.585+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:23.586069+0000) 2024-02-19T20:09:23.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:23.814+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:23.815803+0000) 2024-02-19T20:09:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:24.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:24.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:24.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:24.057+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:24.068+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:24.069080+0000) 2024-02-19T20:09:24.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:24.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:24.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:24.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:24.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:24.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:24.636 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:24.406+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:24.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:24.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:24.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:24.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:24.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:24.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:24.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:24.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:24.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:24.587+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:24.588051+0000) 2024-02-19T20:09:24.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:24.816+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:24.817516+0000) 2024-02-19T20:09:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:25.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:25.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:25.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:25.040+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:25.070+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:25.070717+0000) 2024-02-19T20:09:25.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:25.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:25.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:25.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:25.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:25.440+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:25.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:25.441+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:25.589 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:25.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:25.589 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:25.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:25.589 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:25.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:25.590 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:25.333+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:25.590 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:25.588+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:25.588202+0000) 2024-02-19T20:09:25.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:25.817+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:25.817678+0000) 2024-02-19T20:09:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:26.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:26.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:26.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:26.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:26.020+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:26.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:26.070+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:26.070961+0000) 2024-02-19T20:09:26.588 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:26.429+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55179 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:26.589 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:26.426+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:26.427936+0000) 2024-02-19T20:09:26.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:26.427+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:26.428832+0000) 2024-02-19T20:09:26.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:26.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:26.590 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:26.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:26.590 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:26.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:26.590 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:26.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:26.590 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:26.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:26.590 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:26.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:26.590 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:26.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:26.590 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:26.303+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:26.591 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:26.426+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:26.427427+0000) 2024-02-19T20:09:26.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:26.587+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:26.588343+0000) 2024-02-19T20:09:26.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:26.816+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:26.817839+0000) 2024-02-19T20:09:27.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:27.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:27.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:27.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:27.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:27.047+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:27.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:27.048+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:27.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:27.070+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:27.071204+0000) 2024-02-19T20:09:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:27.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:27.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:27.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:27.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:27.418+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:27.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:27.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:27.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:27.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:27.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:27.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:27.278+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:27.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:27.819+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:27.820351+0000) 2024-02-19T20:09:27.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:27.591+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:27.592003+0000) 2024-02-19T20:09:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:28.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:28.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:28.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:28.068+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:28.072+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:28.073151+0000) 2024-02-19T20:09:28.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:28.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:28.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:28.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:28.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:28.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:28.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:28.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:28.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:28.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:28.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:28.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:28.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:28.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:28.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:28.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:28.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:28.821+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:28.822357+0000) 2024-02-19T20:09:28.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:28.594+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:28.595678+0000) 2024-02-19T20:09:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:29.074+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:29.075101+0000) 2024-02-19T20:09:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:29.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:29.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:29.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:29.111+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:29.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:29.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:29.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:29.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:29.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:29.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:29.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:29.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:29.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:29.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:29.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:29.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:29.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:29.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:29.599 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:29.359+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:29.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:29.596+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:29.597349+0000) 2024-02-19T20:09:29.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:29.822+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:29.823918+0000) 2024-02-19T20:09:30.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:30.075+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:30.076649+0000) 2024-02-19T20:09:30.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:30.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:30.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:30.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:30.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:30.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:30.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:30.106+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:30.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:30.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:30.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:30.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:30.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:30.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:30.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:30.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:30.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:30.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:30.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:30.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:30.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:30.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:30.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:30.350+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:30.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:30.596+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:30.597567+0000) 2024-02-19T20:09:31.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:30.823+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:30.824096+0000) 2024-02-19T20:09:31.331 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:31.075+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:31.076864+0000) 2024-02-19T20:09:31.331 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:31.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:31.331 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:31.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:31.331 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:31.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:31.332 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:31.132+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:31.598 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:31.436+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55218 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:31.598 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:31.434+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:31.435297+0000) 2024-02-19T20:09:31.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:31.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:31.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:31.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:31.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:31.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:31.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:31.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:31.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:31.433+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:31.434771+0000) 2024-02-19T20:09:31.599 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:31.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:31.600 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:31.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:31.600 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:31.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:31.600 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:31.330+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:31.600 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:31.434+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:31.435216+0000) 2024-02-19T20:09:31.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:31.596+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:31.597762+0000) 2024-02-19T20:09:31.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:31.823+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:31.824279+0000) 2024-02-19T20:09:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:32.076+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:32.077474+0000) 2024-02-19T20:09:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:32.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:32.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:32.153+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:32.154+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:32.600 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:32.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:32.600 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:32.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:32.600 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:32.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:32.601 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:32.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:32.601 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:32.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:32.601 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:32.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:32.601 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:32.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:32.601 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:32.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:32.913 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:32.825+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:32.826585+0000) 2024-02-19T20:09:32.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:32.598+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:32.599971+0000) 2024-02-19T20:09:33.281 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:33.080+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:33.081128+0000) 2024-02-19T20:09:33.281 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:33.134+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:33.281 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:33.134+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:33.281 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:33.134+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:33.281 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:33.135+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:33.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:33.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:33.305+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:33.305+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:33.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:33.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:33.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:33.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:33.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:33.601+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:33.602189+0000) 2024-02-19T20:09:33.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:33.827+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:33.828721+0000) 2024-02-19T20:09:34.270 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:34.082+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:34.083212+0000) 2024-02-19T20:09:34.270 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:34.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:34.270 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:34.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:34.270 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:34.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:34.270 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:34.139+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:34.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:34.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:34.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:34.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:34.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:34.268+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:34.269+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:34.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:34.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:34.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:34.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:34.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:34.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:34.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:34.313+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:34.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:34.603+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:34.604168+0000) 2024-02-19T20:09:34.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:34.829+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:34.830278+0000) 2024-02-19T20:09:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:35.084+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:35.085045+0000) 2024-02-19T20:09:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:35.154+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:35.155+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:35.155+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:35.155+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:35.605 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:35.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:35.605 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:35.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:35.605 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:35.331+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:35.605 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:35.332+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:35.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:35.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:35.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:35.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:35.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:35.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:35.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:35.316+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:35.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:35.829+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:35.830426+0000) 2024-02-19T20:09:35.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:35.603+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:35.604348+0000) 2024-02-19T20:09:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:36.084+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:36.085286+0000) 2024-02-19T20:09:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:36.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:36.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:36.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:36.113+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:36.605 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:36.442+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55257 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:36.605 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:36.440+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:36.441779+0000) 2024-02-19T20:09:36.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:36.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:36.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:36.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:36.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:36.334+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:36.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:36.334+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:36.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:36.439+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:36.440833+0000) 2024-02-19T20:09:36.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:36.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:36.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:36.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:36.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:36.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:36.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:36.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:36.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:36.440+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:36.441173+0000) 2024-02-19T20:09:36.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:36.603+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:36.604589+0000) 2024-02-19T20:09:36.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:36.829+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:36.830624+0000) 2024-02-19T20:09:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:37.084+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:37.085458+0000) 2024-02-19T20:09:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:37.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:37.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:37.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:37.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:37.089+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:37.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:37.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:37.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:37.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:37.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:37.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:37.608 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:37.336+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:37.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:37.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:37.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:37.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:37.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:37.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:37.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:37.363+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:37.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:37.831+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:37.832686+0000) 2024-02-19T20:09:37.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:37.606+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:37.607150+0000) 2024-02-19T20:09:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:38.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:38.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:38.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:38.066+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:38.086+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:38.087483+0000) 2024-02-19T20:09:38.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:38.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:38.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:38.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:38.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:38.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:38.610 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:38.348+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:38.610 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:38.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:38.611 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:38.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:38.611 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:38.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:38.611 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:38.322+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:38.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:38.608+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:38.609627+0000) 2024-02-19T20:09:38.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:38.833+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:38.834763+0000) 2024-02-19T20:09:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:39.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:39.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:39.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:39.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:39.025+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:39.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:39.089+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:39.090135+0000) 2024-02-19T20:09:39.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:39.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:39.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:39.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:39.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:39.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:39.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:39.299+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:39.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:39.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:39.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:39.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:39.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:39.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:39.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:39.330+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:39.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:39.611+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:39.612392+0000) 2024-02-19T20:09:39.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:39.835+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:39.836719+0000) 2024-02-19T20:09:40.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:40.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:40.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:40.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:40.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:40.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:40.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:40.026+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:40.182 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:40.090+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:40.091716+0000) 2024-02-19T20:09:40.462 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:40.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:40.463 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:40.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:40.463 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:40.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:40.463 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:40.324+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:40.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:40.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:40.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:40.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:40.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:40.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:40.463 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:40.327+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:40.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:40.611+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:40.612586+0000) 2024-02-19T20:09:41.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:40.835+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:40.836905+0000) 2024-02-19T20:09:41.092 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:41.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:41.092 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:41.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:41.093 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:41.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:41.093 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:41.076+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:41.372 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:41.090+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:41.091887+0000) 2024-02-19T20:09:41.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:41.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:41.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:41.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:41.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:41.319+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:41.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:41.320+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:41.638 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:41.449+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55296 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:41.638 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:41.447+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:41.448325+0000) 2024-02-19T20:09:41.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:41.447+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:41.448220+0000) 2024-02-19T20:09:41.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:41.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:41.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:41.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:41.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:41.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:41.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:41.371+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:41.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:41.446+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:41.447429+0000) 2024-02-19T20:09:41.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:41.611+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:41.612794+0000) 2024-02-19T20:09:41.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:41.836+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:41.837068+0000) 2024-02-19T20:09:42.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:42.091+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:42.092066+0000) 2024-02-19T20:09:42.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:42.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:42.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:42.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:42.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:42.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:42.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:42.113+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:42.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:42.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:42.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:42.340+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:42.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:42.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:42.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:42.281+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:42.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:42.613+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:42.614747+0000) 2024-02-19T20:09:42.915 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:42.838+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:42.839105+0000) 2024-02-19T20:09:43.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:43.093+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:43.094092+0000) 2024-02-19T20:09:43.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:43.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:43.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:43.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:43.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:43.095+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:43.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:43.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:43.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:43.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:43.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:43.250+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:43.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:43.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:43.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:43.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:43.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:43.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:43.840+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:43.841518+0000) 2024-02-19T20:09:43.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:43.615+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:43.616906+0000) 2024-02-19T20:09:44.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:44.096+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:44.097184+0000) 2024-02-19T20:09:44.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:44.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:44.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:44.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:44.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:44.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:44.240 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:44.116+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:44.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:44.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:44.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:44.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:44.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:44.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:44.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:44.238+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:44.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:44.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:44.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:44.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:44.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:44.842+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:44.843658+0000) 2024-02-19T20:09:44.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:44.617+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:44.618478+0000) 2024-02-19T20:09:45.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:45.098+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:45.099828+0000) 2024-02-19T20:09:45.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:45.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:45.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:45.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:45.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:45.150+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:45.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:45.151+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:45.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:45.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:45.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:45.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:45.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:45.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:45.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:45.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:45.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:45.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:45.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:45.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:45.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:45.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:45.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:45.617+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:45.618664+0000) 2024-02-19T20:09:45.915 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:45.842+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:45.843847+0000) 2024-02-19T20:09:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:46.099+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:46.100027+0000) 2024-02-19T20:09:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:46.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:46.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:46.125+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:46.126+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:46.576 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:46.456+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55335 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:46.576 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:46.454+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:46.455030+0000) 2024-02-19T20:09:46.577 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:46.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:46.577 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:46.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:46.577 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:46.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:46.577 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:46.409+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:46.577 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:46.454+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:46.455365+0000) 2024-02-19T20:09:46.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:46.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:46.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:46.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:46.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:46.291+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:46.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:46.292+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:46.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:46.453+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:46.454977+0000) 2024-02-19T20:09:46.844 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:46.617+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:46.618836+0000) 2024-02-19T20:09:47.100 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:46.843+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:46.844032+0000) 2024-02-19T20:09:47.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:47.099+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:47.100219+0000) 2024-02-19T20:09:47.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:47.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:47.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:47.162+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:47.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:47.163+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:47.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:47.163+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:47.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:47.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:47.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:47.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:47.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:47.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:47.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:47.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:47.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:47.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:47.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:47.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:47.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:47.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:47.619+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:47.620891+0000) 2024-02-19T20:09:48.104 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:47.845+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:47.846218+0000) 2024-02-19T20:09:48.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:48.102+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:48.103752+0000) 2024-02-19T20:09:48.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:48.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:48.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:48.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:48.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:48.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:48.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:48.137+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:48.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:48.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:48.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:48.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:48.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:48.314+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:48.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:48.314+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:48.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:48.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:48.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:48.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:48.453+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:48.622+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:48.623036+0000) 2024-02-19T20:09:49.105 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:48.847+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:48.848224+0000) 2024-02-19T20:09:49.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:49.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:49.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:49.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:49.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:49.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:49.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:49.104+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:49.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:49.106+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:49.107280+0000) 2024-02-19T20:09:49.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:49.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:49.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:49.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:49.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:49.286+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:49.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:49.287+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:49.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:49.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:49.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:49.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:49.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:49.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:49.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:49.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:49.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:49.624+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:49.625880+0000) 2024-02-19T20:09:49.973 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:49.848+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:49.849946+0000) 2024-02-19T20:09:50.250 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:50.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:50.250 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:50.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:50.250 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:50.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:50.250 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:50.076+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:50.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:50.108+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:50.109987+0000) 2024-02-19T20:09:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:50.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:50.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:50.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:50.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:50.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:50.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:50.248+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:50.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:50.249+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:50.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:50.625+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:50.626062+0000) 2024-02-19T20:09:50.916 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:50.849+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:50.850122+0000) 2024-02-19T20:09:51.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:51.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:51.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:51.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:51.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:51.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:51.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:51.090+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:51.290 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:51.109+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:51.110174+0000) 2024-02-19T20:09:51.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:51.463+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55374 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:51.460+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:51.461663+0000) 2024-02-19T20:09:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:51.460+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:51.461728+0000) 2024-02-19T20:09:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:51.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:51.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:51.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:51.472+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:51.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:51.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:51.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:51.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:51.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:51.288+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:51.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:51.289+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:51.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:51.461+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:51.462018+0000) 2024-02-19T20:09:51.850 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:51.625+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:51.626217+0000) 2024-02-19T20:09:52.110 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:52.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:52.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:52.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:52.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:52.058+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:52.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:52.059+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:52.111 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:51.849+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:51.850276+0000) 2024-02-19T20:09:52.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:52.109+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:52.110372+0000) 2024-02-19T20:09:52.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:52.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:52.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:52.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:52.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:52.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:52.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:52.248+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:52.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:52.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:52.490+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:52.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:52.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:52.627+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:52.628813+0000) 2024-02-19T20:09:53.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:52.852+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:52.853574+0000) 2024-02-19T20:09:53.113 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:53.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:53.113 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:53.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:53.113 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:53.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:53.113 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:53.077+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:53.449 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:53.111+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:53.112671+0000) 2024-02-19T20:09:53.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:53.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:53.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:53.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:53.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:53.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:53.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:53.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:53.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:53.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:53.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:53.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:53.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:53.631+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:53.632275+0000) 2024-02-19T20:09:54.115 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:53.855+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:53.856016+0000) 2024-02-19T20:09:54.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:54.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:54.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:54.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:54.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:54.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:54.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:54.076+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:54.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:54.114+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:54.115114+0000) 2024-02-19T20:09:54.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:54.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:54.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:54.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:54.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:54.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:54.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:54.244+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:54.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:54.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:54.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:54.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:54.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:54.632+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:54.633863+0000) 2024-02-19T20:09:55.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:54.856+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:54.857588+0000) 2024-02-19T20:09:55.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:55.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:55.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:55.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:55.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:55.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:55.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:55.057+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:55.472 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:55.116+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:55.117934+0000) 2024-02-19T20:09:55.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:55.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:55.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:55.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:55.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:55.231+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:55.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:55.231+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:55.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:55.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:55.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:55.508+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:55.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:55.633+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:55.634051+0000) 2024-02-19T20:09:56.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:55.856+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:55.857774+0000) 2024-02-19T20:09:56.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:56.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:56.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:56.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:56.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:56.032+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:56.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:56.033+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:56.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:56.117+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:56.118109+0000) 2024-02-19T20:09:56.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:56.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:56.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:56.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:56.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:56.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:56.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:56.236+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:56.468+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:56.469048+0000) 2024-02-19T20:09:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:56.466+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:56.467990+0000) 2024-02-19T20:09:56.793 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:09:56.470+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55413 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:09:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:56.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:56.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:56.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:56.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:56.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:56.467+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:56.468617+0000) 2024-02-19T20:09:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:56.633+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:56.634215+0000) 2024-02-19T20:09:57.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:56.856+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:56.857977+0000) 2024-02-19T20:09:57.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:57.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:57.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:57.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:57.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:57.060+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:57.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:57.061+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:57.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:57.117+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:57.118288+0000) 2024-02-19T20:09:57.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:57.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:57.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:57.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:57.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:57.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:57.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:57.256+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:57.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:57.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:57.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:57.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:57.636+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:57.637613+0000) 2024-02-19T20:09:58.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:57.859+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:57.860040+0000) 2024-02-19T20:09:58.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:58.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:58.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:58.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:58.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:58.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:58.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:58.045+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:58.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:58.120+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:58.121880+0000) 2024-02-19T20:09:58.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:58.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:58.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:58.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:58.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:58.262+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:58.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:58.263+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:58.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:58.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:58.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:58.506+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:58.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:58.638+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:58.639782+0000) 2024-02-19T20:09:59.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:58.861+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:58.862448+0000) 2024-02-19T20:09:59.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:59.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:59.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:59.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:59.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:59.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:09:59.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:59.010+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:59.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:09:59.123+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:59.124283+0000) 2024-02-19T20:09:59.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:59.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:59.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:59.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:59.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:59.284+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:09:59.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:59.285+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:09:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:59.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:59.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:59.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:09:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:59.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:09:59.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:09:59.641+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:59.642661+0000) 2024-02-19T20:10:00.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:09:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:09:59.863+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:09:59.864246+0000) 2024-02-19T20:10:00.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:00.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:00.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:00.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:00.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:00.009+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:00.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:00.010+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:00.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:00.125+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:00.126235+0000) 2024-02-19T20:10:00.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:00.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:00.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:00.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:00.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:00.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:00.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:00.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:00.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:00.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:00.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:00.543+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:00.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:00.641+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:00.642865+0000) 2024-02-19T20:10:01.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:00.863+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:00.864418+0000) 2024-02-19T20:10:01.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:01.003+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:01.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:01.003+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:01.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:01.003+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:01.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:01.004+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:01.472 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:01.125+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:01.126381+0000) 2024-02-19T20:10:01.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:01.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:01.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:01.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:01.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:01.223+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:01.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:01.224+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:01.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:01.474+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:01.475473+0000) 2024-02-19T20:10:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:01.474+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:01.475848+0000) 2024-02-19T20:10:01.793 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:01.477+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55452 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:01.474+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:01.475351+0000) 2024-02-19T20:10:01.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:01.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:01.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:01.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:01.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:01.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:01.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:01.579+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:01.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:01.642+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:01.643049+0000) 2024-02-19T20:10:02.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:01.863+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:01.864649+0000) 2024-02-19T20:10:02.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:02.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:02.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:02.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:02.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:02.038+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:02.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:02.039+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:02.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:02.125+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:02.126568+0000) 2024-02-19T20:10:02.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:02.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:02.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:02.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:02.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:02.263+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:02.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:02.263+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:02.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:02.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:02.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:02.616+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:02.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:02.643+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:02.645012+0000) 2024-02-19T20:10:03.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:02.867+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:02.868167+0000) 2024-02-19T20:10:03.120 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:03.046+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:03.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:03.046+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:03.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:03.046+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:03.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:03.046+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:03.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:03.127+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:03.128514+0000) 2024-02-19T20:10:03.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:03.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:03.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:03.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:03.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:03.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:03.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:03.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:03.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:03.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:03.587+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:03.587+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:03.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:03.646+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:03.647064+0000) 2024-02-19T20:10:04.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:03.869+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:03.870220+0000) 2024-02-19T20:10:04.131 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:04.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:04.131 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:04.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:04.131 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:04.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:04.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:04.005+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:04.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:04.129+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:04.130591+0000) 2024-02-19T20:10:04.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:04.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:04.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:04.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:04.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:04.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:04.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:04.255+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:04.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:04.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:04.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:04.559+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:04.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:04.647+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:04.648711+0000) 2024-02-19T20:10:05.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:04.871+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:04.872116+0000) 2024-02-19T20:10:05.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:05.037+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:05.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:05.037+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:05.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:05.037+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:05.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:05.038+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:05.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:05.132+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:05.133171+0000) 2024-02-19T20:10:05.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:05.294+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:05.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:05.294+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:05.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:05.294+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:05.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:05.295+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:05.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:05.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:05.519+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:05.520+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:05.647+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:05.648891+0000) 2024-02-19T20:10:06.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:05.871+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:05.872292+0000) 2024-02-19T20:10:06.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:06.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:06.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:06.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:06.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:06.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:06.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:06.046+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:06.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:06.132+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:06.133339+0000) 2024-02-19T20:10:06.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:06.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:06.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:06.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:06.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:06.337+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:06.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:06.338+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:06.481+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:06.482288+0000) 2024-02-19T20:10:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:06.482+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:06.483138+0000) 2024-02-19T20:10:06.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:06.484+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55491 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:06.481+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:06.482294+0000) 2024-02-19T20:10:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:06.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:06.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:06.539+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:06.540+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:06.648+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:06.649064+0000) 2024-02-19T20:10:07.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:06.871+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:06.872479+0000) 2024-02-19T20:10:07.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:07.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:07.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:07.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.020+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:07.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.132+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:07.133498+0000) 2024-02-19T20:10:07.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:07.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:07.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:07.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:07.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:07.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:07.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:07.367+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:07.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:07.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:07.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:07.549+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:07.651+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:07.652521+0000) 2024-02-19T20:10:08.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:07.873+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:07.874578+0000) 2024-02-19T20:10:08.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:08.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:08.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:08.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:07.991+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:08.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:08.134+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:08.135619+0000) 2024-02-19T20:10:08.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:08.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:08.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:08.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:08.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:08.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:08.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:08.333+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:08.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:08.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:08.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:08.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:08.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:08.653+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:08.654624+0000) 2024-02-19T20:10:09.138 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:08.875+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:08.876626+0000) 2024-02-19T20:10:09.138 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:08.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:09.139 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:08.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:09.139 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:08.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:09.139 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:08.948+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:09.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:09.137+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:09.138192+0000) 2024-02-19T20:10:09.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:09.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:09.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:09.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:09.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:09.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:09.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:09.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:09.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:09.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:09.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:09.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:09.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:09.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:09.655+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:09.656299+0000) 2024-02-19T20:10:10.140 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:09.877+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:09.878259+0000) 2024-02-19T20:10:10.140 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:09.994+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:10.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:09.994+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:10.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:09.994+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:10.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:09.994+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:10.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:10.138+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:10.139957+0000) 2024-02-19T20:10:10.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:10.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:10.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:10.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:10.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:10.401+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:10.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:10.402+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:10.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:10.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:10.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:10.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:10.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:10.549+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:10.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:10.550+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:10.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:10.655+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:10.656471+0000) 2024-02-19T20:10:10.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:10.877+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:10.878469+0000) 2024-02-19T20:10:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:11.027+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:11.027+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:11.027+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:11.027+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:11.139+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:11.140125+0000) 2024-02-19T20:10:11.610 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:11.491+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55530 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:11.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:11.489+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:11.490626+0000) 2024-02-19T20:10:11.611 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:11.488+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:11.489683+0000) 2024-02-19T20:10:11.611 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:11.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:11.611 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:11.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:11.611 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:11.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:11.611 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:11.513+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:11.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:11.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:11.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:11.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:11.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:11.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:11.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:11.358+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:11.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:11.488+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:11.489029+0000) 2024-02-19T20:10:11.879 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:11.655+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:11.656650+0000) 2024-02-19T20:10:12.140 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:11.877+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:11.878735+0000) 2024-02-19T20:10:12.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:12.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:12.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:12.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.021+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:12.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.139+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:12.140282+0000) 2024-02-19T20:10:12.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:12.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:12.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:12.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:12.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:12.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:12.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:12.535+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:12.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:12.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:12.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:12.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:12.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:12.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:12.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:12.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:12.917 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:12.660+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:12.660124+0000) 2024-02-19T20:10:12.918 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:12.881+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:12.881362+0000) 2024-02-19T20:10:13.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:13.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:13.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:12.991+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:13.142+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:13.142497+0000) 2024-02-19T20:10:13.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:13.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:13.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:13.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:13.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:13.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:13.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:13.581+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:13.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:13.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:13.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:13.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:13.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:13.395+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:13.664 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:13.396+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:13.918 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:13.883+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:13.884954+0000) 2024-02-19T20:10:13.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:13.661+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:13.662309+0000) 2024-02-19T20:10:14.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.025+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:14.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.025+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:14.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.025+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.026+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.143+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:14.144635+0000) 2024-02-19T20:10:14.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:14.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:14.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:14.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:14.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:14.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:14.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:14.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:14.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:14.663+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:14.664431+0000) 2024-02-19T20:10:14.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:14.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:14.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:14.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:14.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:14.416+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:14.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:14.416+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:14.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:14.886+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:14.887906+0000) 2024-02-19T20:10:15.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:15.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:15.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:14.988+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:15.145+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:15.146598+0000) 2024-02-19T20:10:15.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:15.594+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:15.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:15.594+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:15.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:15.594+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:15.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:15.595+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:15.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:15.663+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:15.664620+0000) 2024-02-19T20:10:15.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:15.463+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:15.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:15.463+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:15.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:15.463+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:15.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:15.464+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:16.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:15.887+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:15.888169+0000) 2024-02-19T20:10:16.147 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:15.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:16.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:15.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:16.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:15.960+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:16.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:15.960+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:16.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:16.145+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:16.146818+0000) 2024-02-19T20:10:16.701 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:16.498+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55569 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:16.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:16.495+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:16.496608+0000) 2024-02-19T20:10:16.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:16.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:16.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:16.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:16.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:16.428+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:16.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:16.429+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:16.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:16.495+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:16.496834+0000) 2024-02-19T20:10:16.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:16.495+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:16.496951+0000) 2024-02-19T20:10:16.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:16.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:16.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:16.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:16.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:16.551+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:16.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:16.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:16.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:16.663+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:16.664821+0000) 2024-02-19T20:10:16.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:16.887+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:16.888387+0000) 2024-02-19T20:10:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:16.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:16.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:16.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:16.991+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:17.145+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:17.146990+0000) 2024-02-19T20:10:17.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:17.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:17.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:17.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:17.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:17.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:17.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:17.573+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:17.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:17.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:17.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:17.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:17.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:17.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:17.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:17.410+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:17.918 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:17.889+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:17.890377+0000) 2024-02-19T20:10:17.918 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:17.666+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:17.667214+0000) 2024-02-19T20:10:18.238 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:17.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:18.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:17.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:18.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:17.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:18.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:17.946+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:18.239 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:18.149+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:18.150335+0000) 2024-02-19T20:10:18.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:18.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:18.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:18.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:18.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:18.369+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:18.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:18.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:18.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:18.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:18.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:18.580+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:18.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:18.581+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:18.893 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:18.669+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:18.670399+0000) 2024-02-19T20:10:19.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:18.891+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:18.892425+0000) 2024-02-19T20:10:19.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:18.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:19.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:18.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:19.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:18.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:19.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:18.990+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:19.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:19.151+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:19.152485+0000) 2024-02-19T20:10:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:19.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:19.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:19.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:19.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:19.356+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:19.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:19.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:19.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:19.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:19.600+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:19.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:19.671+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:19.672053+0000) 2024-02-19T20:10:20.154 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:19.893+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:19.894300+0000) 2024-02-19T20:10:20.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:19.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:20.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:19.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:20.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:19.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:20.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:19.970+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:20.153+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:20.154456+0000) 2024-02-19T20:10:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:20.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:20.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:20.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:20.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:20.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:20.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:20.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:20.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:20.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:20.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:20.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:20.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:20.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:20.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:20.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:20.671+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:20.672256+0000) 2024-02-19T20:10:21.155 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:20.893+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:20.894491+0000) 2024-02-19T20:10:21.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:20.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:21.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:20.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:21.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:20.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:21.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:20.924+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:21.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:21.153+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:21.154651+0000) 2024-02-19T20:10:21.701 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:21.504+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55608 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:21.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:21.501+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:21.502901+0000) 2024-02-19T20:10:21.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:21.502+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:21.503599+0000) 2024-02-19T20:10:21.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:21.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:21.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:21.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:21.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:21.616+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:21.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:21.616+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:21.703 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:21.671+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:21.672451+0000) 2024-02-19T20:10:21.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:21.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:21.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:21.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:21.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:21.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:21.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:21.425+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:21.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:21.502+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:21.503579+0000) 2024-02-19T20:10:21.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:21.893+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:21.894680+0000) 2024-02-19T20:10:21.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:21.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:21.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:21.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:21.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:21.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:21.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:21.942+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:22.241 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:22.153+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:22.154873+0000) 2024-02-19T20:10:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:22.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:22.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:22.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:22.392+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:22.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:22.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:22.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:22.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:22.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:22.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:22.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:22.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:22.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:22.673+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:22.675000+0000) 2024-02-19T20:10:23.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:22.895+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:22.896697+0000) 2024-02-19T20:10:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:22.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:22.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:22.908+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:22.909+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:23.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:23.156+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:23.157320+0000) 2024-02-19T20:10:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:23.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:23.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:23.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:23.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:23.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:23.899 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:23.620+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:23.899 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:23.620+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:23.899 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:23.620+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:23.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:23.621+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:23.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:23.677+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:23.678618+0000) 2024-02-19T20:10:24.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:23.898+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:23.899145+0000) 2024-02-19T20:10:24.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:23.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:24.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:23.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:24.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:23.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:24.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:23.905+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:24.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:24.158+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:24.159485+0000) 2024-02-19T20:10:24.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:24.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:24.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:24.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:24.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:24.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:24.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:24.403+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:24.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:24.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:24.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:24.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:24.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:24.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:24.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:24.584+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:24.881 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:24.679+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:24.680379+0000) 2024-02-19T20:10:25.161 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:24.900+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:24.901544+0000) 2024-02-19T20:10:25.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:24.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:25.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:24.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:25.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:24.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:25.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:24.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:25.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:25.160+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:25.161237+0000) 2024-02-19T20:10:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:25.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:25.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:25.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:25.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:25.381+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:25.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:25.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:25.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:25.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:25.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:25.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:25.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:25.583+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:25.902 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:25.679+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:25.680601+0000) 2024-02-19T20:10:26.161 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:25.900+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:25.901721+0000) 2024-02-19T20:10:26.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:25.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:26.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:25.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:26.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:25.920+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:26.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:25.921+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:26.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:26.160+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:26.161459+0000) 2024-02-19T20:10:26.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:26.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:26.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:26.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:26.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:26.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:26.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:26.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:26.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:26.510+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:26.511388+0000) 2024-02-19T20:10:26.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:26.511+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55647 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:26.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:26.509+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:26.510133+0000) 2024-02-19T20:10:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:26.509+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:26.510189+0000) 2024-02-19T20:10:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:26.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:26.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:26.603+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:26.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:26.603+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:26.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:26.679+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:26.680783+0000) 2024-02-19T20:10:27.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:26.900+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:26.901912+0000) 2024-02-19T20:10:27.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:26.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:27.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:26.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:27.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:26.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:27.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:26.875+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:27.160+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:27.161652+0000) 2024-02-19T20:10:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:27.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:27.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:27.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:27.393+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:27.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:27.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:27.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:27.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:27.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:27.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:27.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:27.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:27.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:27.683+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:27.684464+0000) 2024-02-19T20:10:28.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:27.904+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:27.905375+0000) 2024-02-19T20:10:28.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:27.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:28.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:27.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:28.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:27.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:28.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:27.853+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:28.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:28.164+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:28.165125+0000) 2024-02-19T20:10:28.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:28.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:28.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:28.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:28.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:28.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:28.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:28.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:28.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:28.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:28.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:28.536+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:28.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:28.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:28.687+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:28.688109+0000) 2024-02-19T20:10:29.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:28.907+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:28.908841+0000) 2024-02-19T20:10:29.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:28.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:29.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:28.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:29.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:28.855+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:29.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:28.856+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:29.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:29.166+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:29.167178+0000) 2024-02-19T20:10:29.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:29.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:29.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:29.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:29.365+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:29.365+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:29.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:29.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:29.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:29.562+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:29.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:29.690+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:29.691154+0000) 2024-02-19T20:10:30.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:29.910+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:29.911642+0000) 2024-02-19T20:10:30.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:29.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:30.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:29.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:30.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:29.896+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:30.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:29.896+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:30.167+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:30.168812+0000) 2024-02-19T20:10:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:30.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:30.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:30.392+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:30.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:30.392+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:30.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:30.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:30.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:30.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:30.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:30.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:30.900 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:30.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:30.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:30.690+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:30.691314+0000) 2024-02-19T20:10:31.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:30.910+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:30.911841+0000) 2024-02-19T20:10:31.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:30.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:31.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:30.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:31.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:30.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:31.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:30.899+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:31.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:31.168+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:31.169032+0000) 2024-02-19T20:10:31.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:31.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:31.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:31.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:31.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:31.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:31.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:31.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:31.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:31.516+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:31.517568+0000) 2024-02-19T20:10:31.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:31.518+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55686 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:31.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:31.516+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:31.517487+0000) 2024-02-19T20:10:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:31.516+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:31.517473+0000) 2024-02-19T20:10:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:31.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:31.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:31.568+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:31.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:31.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:31.690+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:31.691491+0000) 2024-02-19T20:10:32.169 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:31.911+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:31.912066+0000) 2024-02-19T20:10:32.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:31.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:32.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:31.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:32.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:31.940+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:32.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:31.940+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:32.168+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:32.169314+0000) 2024-02-19T20:10:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:32.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:32.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:32.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:32.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:32.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:32.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:32.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:32.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:32.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:32.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:32.617+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:32.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:32.618+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:32.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:32.692+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:32.693595+0000) 2024-02-19T20:10:33.172 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:32.914+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:32.915633+0000) 2024-02-19T20:10:33.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:32.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:33.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:32.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:33.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:32.913+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:33.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:32.914+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:33.171+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:33.172582+0000) 2024-02-19T20:10:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:33.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:33.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:33.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:33.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:33.357+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:33.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:33.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:33.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:33.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:33.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:33.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:33.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:33.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:33.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:33.695+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:33.696156+0000) 2024-02-19T20:10:34.175 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:33.916+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:33.917740+0000) 2024-02-19T20:10:34.175 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:33.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:34.175 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:33.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:34.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:33.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:34.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:33.877+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:34.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:34.173+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:34.174881+0000) 2024-02-19T20:10:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:34.390+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:34.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:34.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:34.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:34.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:34.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:34.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:34.894 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:34.584+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:34.895 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:34.696+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:34.697838+0000) 2024-02-19T20:10:35.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:34.918+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:34.919407+0000) 2024-02-19T20:10:35.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:34.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:35.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:34.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:35.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:34.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:35.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:34.893+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:35.176+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:35.177621+0000) 2024-02-19T20:10:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:35.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:35.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:35.353+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:35.353+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:35.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:35.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:35.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:35.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:35.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:35.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:35.899 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:35.551+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:35.899 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:35.696+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:35.698012+0000) 2024-02-19T20:10:36.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:35.918+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:35.919613+0000) 2024-02-19T20:10:36.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:35.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:36.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:35.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:36.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:35.897+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:36.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:35.898+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:36.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:36.176+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:36.177811+0000) 2024-02-19T20:10:36.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:36.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:36.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:36.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:36.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:36.356+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:36.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:36.356+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:36.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:36.523+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:36.524788+0000) 2024-02-19T20:10:36.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:36.525+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55725 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:36.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:36.523+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:36.524192+0000) 2024-02-19T20:10:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:36.524+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:36.525239+0000) 2024-02-19T20:10:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:36.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:36.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:36.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:36.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:36.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:36.697+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:36.698168+0000) 2024-02-19T20:10:37.178 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:36.918+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:36.919806+0000) 2024-02-19T20:10:37.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:36.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:37.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:36.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:37.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:36.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:37.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:36.902+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:37.176+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:37.177985+0000) 2024-02-19T20:10:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:37.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:37.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:37.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:37.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:37.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:37.883 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:37.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:37.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:37.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:37.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:37.576+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:37.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:37.576+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:37.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:37.699+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:37.700184+0000) 2024-02-19T20:10:38.180 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:37.921+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:37.922950+0000) 2024-02-19T20:10:38.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:37.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:38.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:37.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:38.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:37.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:38.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:37.883+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:38.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:38.179+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:38.180236+0000) 2024-02-19T20:10:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:38.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:38.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:38.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:38.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:38.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:38.836 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:38.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:38.836 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:38.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:38.836 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:38.550+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:38.836 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:38.550+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:38.836 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:38.701+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:38.702203+0000) 2024-02-19T20:10:39.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:38.924+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:38.925332+0000) 2024-02-19T20:10:39.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:38.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:39.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:38.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:39.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:38.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:39.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:38.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:39.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:39.181+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:39.182836+0000) 2024-02-19T20:10:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:39.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:39.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:39.357+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:39.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:39.358+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:39.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:39.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:39.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:39.540+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:39.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:39.703+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:39.704353+0000) 2024-02-19T20:10:40.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:39.926+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:39.927545+0000) 2024-02-19T20:10:40.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:39.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:40.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:39.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:40.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:39.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:40.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:39.839+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:40.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:40.183+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:40.184539+0000) 2024-02-19T20:10:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:40.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:40.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:40.367+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:40.368+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:40.840 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:40.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:40.840 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:40.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:40.840 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:40.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:40.840 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:40.542+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:40.841 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:40.703+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:40.704535+0000) 2024-02-19T20:10:41.105 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:40.926+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:40.927750+0000) 2024-02-19T20:10:41.106 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:40.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:41.106 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:40.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:41.106 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:40.839+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:41.106 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:40.840+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:41.392 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:41.183+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:41.184734+0000) 2024-02-19T20:10:41.645 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:41.532+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55764 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:41.645 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:41.529+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:41.530623+0000) 2024-02-19T20:10:41.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:41.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:41.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:41.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:41.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:41.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:41.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:41.392+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:41.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:41.529+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:41.530596+0000) 2024-02-19T20:10:41.646 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:41.530+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:41.531048+0000) 2024-02-19T20:10:41.646 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:41.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:41.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:41.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:41.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:41.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:41.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:41.584+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:41.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:41.703+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:41.704719+0000) 2024-02-19T20:10:41.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:41.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:41.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:41.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:41.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:41.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:41.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:41.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:42.185 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:41.926+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:41.927956+0000) 2024-02-19T20:10:42.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:42.183+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:42.184916+0000) 2024-02-19T20:10:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:42.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:42.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:42.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:42.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:42.431+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:42.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:42.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:42.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:42.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:42.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:42.783+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:42.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:42.784+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:42.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:42.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:42.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:42.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:42.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:42.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:42.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:42.606+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:42.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:42.705+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:42.706800+0000) 2024-02-19T20:10:43.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:42.930+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:42.931523+0000) 2024-02-19T20:10:43.477 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:43.187+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:43.188506+0000) 2024-02-19T20:10:43.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:43.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:43.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:43.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:43.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:43.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:43.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:43.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:43.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:43.709+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:43.710401+0000) 2024-02-19T20:10:43.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:43.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:43.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:43.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:43.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:43.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:43.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:43.477+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:43.932+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:43.933618+0000) 2024-02-19T20:10:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:43.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:43.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:43.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:44.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:43.772+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:44.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:44.189+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:44.191001+0000) 2024-02-19T20:10:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:44.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:44.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:44.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:44.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:44.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:44.712+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:44.713219+0000) 2024-02-19T20:10:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:44.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:44.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:44.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:44.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:44.725+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:44.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:44.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:44.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:44.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:44.499+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:44.794 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:44.500+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:45.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:44.934+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:44.935718+0000) 2024-02-19T20:10:45.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:45.192+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:45.193991+0000) 2024-02-19T20:10:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:45.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:45.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:45.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:45.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:45.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:45.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:45.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:45.529+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:45.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:45.530+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:45.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:45.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:45.585+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:45.586+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:45.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:45.712+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:45.713420+0000) 2024-02-19T20:10:46.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:45.934+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:45.935920+0000) 2024-02-19T20:10:46.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:46.193+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:46.194179+0000) 2024-02-19T20:10:46.778 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:46.539+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55803 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:46.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:46.536+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:46.537980+0000) 2024-02-19T20:10:46.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:46.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:46.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:46.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:46.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:46.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:46.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:46.536+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:46.537913+0000) 2024-02-19T20:10:46.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:46.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:46.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:46.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:46.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:46.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:46.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:46.577+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:46.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:46.712+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:46.713620+0000) 2024-02-19T20:10:46.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:46.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:46.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:46.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:46.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:46.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:46.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:46.482+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:46.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:46.537+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:46.538356+0000) 2024-02-19T20:10:47.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:46.935+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:46.936122+0000) 2024-02-19T20:10:47.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:46.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:47.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:47.193+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:47.194384+0000) 2024-02-19T20:10:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:47.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:47.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:47.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:47.476+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:47.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:47.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:47.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:47.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:47.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:47.565+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:47.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:47.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:47.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:47.716+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:47.717430+0000) 2024-02-19T20:10:48.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:47.937+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:47.938793+0000) 2024-02-19T20:10:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:47.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:47.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:47.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:48.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:47.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:48.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:48.195+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:48.196478+0000) 2024-02-19T20:10:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:48.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:48.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:48.501+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:48.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:48.501+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:48.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:48.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:48.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:48.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:48.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:48.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:48.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:48.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:48.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:48.528+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:48.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:48.720+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:48.721253+0000) 2024-02-19T20:10:49.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:48.940+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:48.941321+0000) 2024-02-19T20:10:49.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:49.197+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:49.198490+0000) 2024-02-19T20:10:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:49.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:49.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:49.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:49.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:49.717+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:49.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:49.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:49.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:49.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:49.527+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:49.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:49.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:49.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:49.514+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:49.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:49.722+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:49.723076+0000) 2024-02-19T20:10:50.186 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:49.942+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:49.943399+0000) 2024-02-19T20:10:50.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:50.199+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:50.200388+0000) 2024-02-19T20:10:50.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:50.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:50.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:50.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:50.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:50.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:50.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:50.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:50.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:50.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:50.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:50.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:50.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:50.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:50.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:50.567+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:50.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:50.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:50.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:50.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:50.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:50.525+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:50.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:50.722+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:50.723249+0000) 2024-02-19T20:10:51.200 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:50.942+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:50.943638+0000) 2024-02-19T20:10:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:51.199+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:51.200583+0000) 2024-02-19T20:10:51.808 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:51.546+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55842 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:51.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:51.543+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:51.544563+0000) 2024-02-19T20:10:51.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:51.807+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:51.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:51.807+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:51.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:51.807+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:51.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:51.544+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:51.545744+0000) 2024-02-19T20:10:51.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:51.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:51.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:51.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:51.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:51.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:51.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:51.559+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:51.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:51.722+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:51.723450+0000) 2024-02-19T20:10:51.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:51.543+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:51.544891+0000) 2024-02-19T20:10:51.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:51.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:51.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:51.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:51.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:51.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:51.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:51.576+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:52.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:51.808+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:52.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:51.942+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:51.943847+0000) 2024-02-19T20:10:52.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:52.199+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:52.200792+0000) 2024-02-19T20:10:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:52.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:52.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:52.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:52.540+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:52.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:52.541+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:52.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:52.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:52.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:52.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:52.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:52.568+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:52.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:52.724+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:52.725506+0000) 2024-02-19T20:10:52.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:52.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:52.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:52.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:52.794 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:52.770+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:52.795 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:52.771+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:53.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:52.944+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:52.945831+0000) 2024-02-19T20:10:53.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:53.201+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:53.202797+0000) 2024-02-19T20:10:53.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:53.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:53.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:53.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:53.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:53.572+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:53.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:53.573+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:53.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:53.727+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:53.728920+0000) 2024-02-19T20:10:53.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:53.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:53.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:53.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:53.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:53.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:53.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:53.792+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:53.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:53.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:53.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:53.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:53.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:53.545+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:53.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:53.546+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:54.205 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:53.946+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:53.947995+0000) 2024-02-19T20:10:54.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:54.203+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:54.204798+0000) 2024-02-19T20:10:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:54.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:54.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:54.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:54.745+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:54.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:54.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:54.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:54.505+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:54.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:54.505+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:54.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:54.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:54.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:54.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:54.571+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:54.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:54.729+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:54.730677+0000) 2024-02-19T20:10:55.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:54.948+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:54.949711+0000) 2024-02-19T20:10:55.495 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:55.205+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:55.206626+0000) 2024-02-19T20:10:55.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:55.494+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:55.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:55.494+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:55.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:55.494+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:55.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:55.495+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:55.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:55.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:55.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:55.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:55.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:55.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:55.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:55.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:55.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:55.729+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:55.730865+0000) 2024-02-19T20:10:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:55.948+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:55.949877+0000) 2024-02-19T20:10:56.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:55.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:56.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:55.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:56.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:55.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:56.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:55.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:56.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:56.205+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:56.206860+0000) 2024-02-19T20:10:56.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:10:56.554+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55881 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:10:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:56.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:56.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:56.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:56.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:56.482+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:56.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:56.551+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:56.552549+0000) 2024-02-19T20:10:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:56.552+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:56.553451+0000) 2024-02-19T20:10:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:56.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:56.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:56.713+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:56.714+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:56.550+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:56.551947+0000) 2024-02-19T20:10:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:56.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:56.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:56.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:56.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:56.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:56.730+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:56.731052+0000) 2024-02-19T20:10:57.207 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:56.949+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:56.950056+0000) 2024-02-19T20:10:57.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:57.206+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:57.207097+0000) 2024-02-19T20:10:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:57.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:57.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:57.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:57.727+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:57.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:57.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:57.528+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:57.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:57.529+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:57.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:57.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:57.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:57.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:57.510+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:57.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:57.732+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:57.733060+0000) 2024-02-19T20:10:58.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:57.951+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:57.952427+0000) 2024-02-19T20:10:58.494 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:58.208+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:58.209612+0000) 2024-02-19T20:10:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:58.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:58.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:58.561+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:58.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:58.562+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:58.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:58.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:58.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:10:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:58.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:58.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:58.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:58.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:58.494+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:10:58.794 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:58.735+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:58.736658+0000) 2024-02-19T20:10:59.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:58.954+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:58.955051+0000) 2024-02-19T20:10:59.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:59.211+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:59.211805+0000) 2024-02-19T20:10:59.733 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:59.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:59.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:59.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:59.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:59.556+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:10:59.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:59.557+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:10:59.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:59.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:59.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:59.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:59.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:59.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:10:59.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:59.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:00.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:10:59.738+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:59.738811+0000) 2024-02-19T20:11:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:10:59.956+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:10:59.956816+0000) 2024-02-19T20:11:00.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:59.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:59.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:59.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:00.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:10:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:10:59.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:00.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:00.213+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:00.213683+0000) 2024-02-19T20:11:00.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:00.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:00.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:00.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:00.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:00.526+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:00.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:00.527+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:00.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:00.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:00.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:00.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:00.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:00.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:00.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:00.695+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:00.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:00.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:00.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:00.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:00.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:00.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:00.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:00.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:00.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:00.738+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:00.739036+0000) 2024-02-19T20:11:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:00.955+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:00.957010+0000) 2024-02-19T20:11:01.428 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:01.212+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:01.214000+0000) 2024-02-19T20:11:01.682 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:01.560+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55920 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:01.682 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:01.559+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:01.560220+0000) 2024-02-19T20:11:01.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:01.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:01.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:01.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:01.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:01.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:01.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:01.520+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:01.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:01.558+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:01.559550+0000) 2024-02-19T20:11:01.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:01.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:01.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:01.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:01.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:01.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:01.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:01.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:01.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:01.558+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:01.559799+0000) 2024-02-19T20:11:01.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:01.738+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:01.739215+0000) 2024-02-19T20:11:01.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:01.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:01.957 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:01.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:01.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:01.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:01.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:01.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:02.214 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:01.956+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:01.957182+0000) 2024-02-19T20:11:02.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:02.213+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:02.214199+0000) 2024-02-19T20:11:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:02.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:02.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:02.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:02.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:02.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:02.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:02.740+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:02.741175+0000) 2024-02-19T20:11:02.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:02.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:02.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:02.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:02.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:02.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:02.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:02.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:02.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:02.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:02.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:02.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:02.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:02.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:02.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:02.565+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:03.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:02.958+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:02.959347+0000) 2024-02-19T20:11:03.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:03.215+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:03.216172+0000) 2024-02-19T20:11:03.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:03.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:03.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:03.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:03.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:03.408+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:03.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:03.408+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:03.742+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:03.743228+0000) 2024-02-19T20:11:03.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:03.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:03.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:03.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:03.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:03.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:03.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:03.531+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:03.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:03.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:03.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:03.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:03.685+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:04.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:03.960+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:03.961434+0000) 2024-02-19T20:11:04.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:04.217+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:04.218675+0000) 2024-02-19T20:11:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:04.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:04.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:04.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:04.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:04.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:04.744+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:04.745016+0000) 2024-02-19T20:11:04.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:04.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:04.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:04.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:04.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:04.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:04.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:04.577+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:04.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:04.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:04.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:04.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:04.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:04.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:04.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:04.705+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:05.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:04.962+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:04.963636+0000) 2024-02-19T20:11:05.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:05.219+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:05.220709+0000) 2024-02-19T20:11:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:05.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:05.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:05.405+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:05.405+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:05.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:05.744+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:05.745208+0000) 2024-02-19T20:11:05.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:05.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:05.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:05.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:05.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:05.692+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:05.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:05.693+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:05.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:05.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:05.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:05.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:05.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:05.596+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:05.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:05.597+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:06.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:05.962+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:05.963822+0000) 2024-02-19T20:11:06.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:06.219+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:06.220905+0000) 2024-02-19T20:11:06.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:06.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:06.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:06.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:06.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:06.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:06.924 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:06.567+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55959 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:06.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:06.565+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:06.566454+0000) 2024-02-19T20:11:06.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:06.744+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:06.745402+0000) 2024-02-19T20:11:06.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:06.565+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:06.566988+0000) 2024-02-19T20:11:06.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:06.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:06.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:06.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:06.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:06.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:06.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:06.703+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:06.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:06.565+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:06.566203+0000) 2024-02-19T20:11:06.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:06.620+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:06.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:06.620+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:06.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:06.620+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:06.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:06.620+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:07.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:06.962+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:06.963991+0000) 2024-02-19T20:11:07.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:07.220+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:07.221126+0000) 2024-02-19T20:11:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:07.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:07.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:07.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:07.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:07.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:07.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:07.746+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:07.747450+0000) 2024-02-19T20:11:07.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:07.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:07.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:07.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:07.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:07.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:07.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:07.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:07.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:07.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:07.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:07.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:07.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:07.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:07.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:07.657+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:08.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:07.965+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:07.966063+0000) 2024-02-19T20:11:08.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:08.222+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:08.223360+0000) 2024-02-19T20:11:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:08.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:08.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:08.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:08.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:08.379+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:08.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:08.749+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:08.750952+0000) 2024-02-19T20:11:08.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:08.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:08.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:08.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:08.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:08.631+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:08.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:08.632+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:08.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:08.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:08.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:08.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:08.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:08.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:08.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:08.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:09.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:08.967+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:08.968206+0000) 2024-02-19T20:11:09.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:09.224+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:09.225505+0000) 2024-02-19T20:11:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:09.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:09.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:09.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:09.382+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:09.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:09.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:09.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:09.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:09.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:09.686+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:09.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:09.687+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:09.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:09.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:09.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:09.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:09.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:09.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:09.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:09.605+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:10.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:09.969+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:09.970135+0000) 2024-02-19T20:11:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:09.751+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:09.752716+0000) 2024-02-19T20:11:10.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:10.227+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:10.228559+0000) 2024-02-19T20:11:10.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:10.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:10.649 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:10.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:10.649 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:10.603+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:10.649 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:10.604+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:10.649 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:10.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:10.649 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:10.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:10.649 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:10.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:10.649 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:10.390+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:10.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:10.751+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:10.752882+0000) 2024-02-19T20:11:10.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:10.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:10.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:10.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:10.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:10.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:10.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:10.683+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:11.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:10.969+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:10.970306+0000) 2024-02-19T20:11:11.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:11.227+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:11.228756+0000) 2024-02-19T20:11:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:11.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:11.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:11.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:11.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:11.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:11.825 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:11.574+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 55998 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:11.825 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:11.572+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:11.573528+0000) 2024-02-19T20:11:11.826 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:11.752+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:11.753127+0000) 2024-02-19T20:11:11.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:11.572+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:11.573018+0000) 2024-02-19T20:11:11.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:11.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:11.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:11.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:11.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:11.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:11.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:11.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:11.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:11.572+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:11.573298+0000) 2024-02-19T20:11:11.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:11.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:11.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:11.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:11.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:11.645+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:11.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:11.646+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:12.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:11.969+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:11.970515+0000) 2024-02-19T20:11:12.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:12.227+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:12.228980+0000) 2024-02-19T20:11:12.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:12.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:12.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:12.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:12.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:12.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:12.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:12.642+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:12.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:12.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:12.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:12.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:12.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:12.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:12.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:12.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:12.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:12.754+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:12.755240+0000) 2024-02-19T20:11:12.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:12.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:12.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:12.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:12.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:12.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:12.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:12.693+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:13.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:12.972+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:12.973158+0000) 2024-02-19T20:11:13.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:13.229+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:13.230930+0000) 2024-02-19T20:11:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:13.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:13.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:13.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:13.347+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:13.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:13.756+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:13.757926+0000) 2024-02-19T20:11:13.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:13.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:13.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:13.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:13.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:13.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:13.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:13.599+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:13.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:13.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:13.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:13.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:13.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:13.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:13.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:13.697+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:14.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:13.974+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:13.975327+0000) 2024-02-19T20:11:14.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:14.232+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:14.233139+0000) 2024-02-19T20:11:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:14.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:14.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:14.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:14.388+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:14.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:14.389+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:14.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:14.760+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:14.761092+0000) 2024-02-19T20:11:14.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:14.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:14.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:14.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:14.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:14.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:14.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:14.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:14.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:14.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:14.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:14.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:14.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:14.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:14.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:14.705+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:15.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:14.976+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:14.977512+0000) 2024-02-19T20:11:15.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:15.234+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:15.235190+0000) 2024-02-19T20:11:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:15.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:15.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:15.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:15.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:15.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:15.356+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:15.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:15.760+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:15.761343+0000) 2024-02-19T20:11:15.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:15.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:15.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:15.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:15.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:15.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:15.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:15.663+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:15.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:15.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:15.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:15.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:15.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:15.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:15.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:15.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:16.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:15.976+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:15.977706+0000) 2024-02-19T20:11:16.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:16.234+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:16.235458+0000) 2024-02-19T20:11:16.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:16.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:16.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:16.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:16.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:16.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:16.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:16.309+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:16.925 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:16.581+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56037 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:16.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:16.580+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:16.581216+0000) 2024-02-19T20:11:16.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:16.760+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:16.761511+0000) 2024-02-19T20:11:16.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:16.578+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:16.579600+0000) 2024-02-19T20:11:16.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:16.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:16.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:16.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:16.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:16.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:16.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:16.655+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:16.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:16.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:16.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:16.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:16.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:16.564+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:16.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:16.565+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:16.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:16.579+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:16.580209+0000) 2024-02-19T20:11:17.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:16.976+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:16.977888+0000) 2024-02-19T20:11:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:17.234+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:17.235614+0000) 2024-02-19T20:11:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:17.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:17.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:17.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:17.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:17.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:17.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:17.762+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:17.763855+0000) 2024-02-19T20:11:17.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:17.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:17.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:17.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:17.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:17.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:17.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:17.582+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:17.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:17.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:17.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:17.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:17.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:17.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:17.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:17.664+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:18.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:17.980+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:17.981362+0000) 2024-02-19T20:11:18.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:18.238+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:18.239018+0000) 2024-02-19T20:11:18.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:18.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:18.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:18.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:18.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:18.297+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:18.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:18.764+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:18.766009+0000) 2024-02-19T20:11:18.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:18.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:18.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:18.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:18.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:18.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:18.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:18.624+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:18.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:18.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:18.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:18.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:18.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:18.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:18.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:18.680+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:19.242 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:18.983+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:18.984803+0000) 2024-02-19T20:11:19.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:19.241+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:19.242480+0000) 2024-02-19T20:11:19.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:19.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:19.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:19.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:19.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:19.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:19.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:19.259+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:19.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:19.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:19.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:19.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:19.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:19.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:19.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:19.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:19.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:19.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:19.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:19.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:19.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:19.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:19.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:19.728+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:20.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:19.985+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:19.986586+0000) 2024-02-19T20:11:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:19.766+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:19.767858+0000) 2024-02-19T20:11:20.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:20.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:20.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:20.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:20.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:20.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:20.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:20.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:20.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:20.243+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:20.244533+0000) 2024-02-19T20:11:20.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:20.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:20.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:20.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:20.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:20.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:20.767+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:20.768047+0000) 2024-02-19T20:11:20.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:20.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:20.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:20.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:20.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:20.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:20.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:20.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:20.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:20.700+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:21.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:20.985+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:20.986775+0000) 2024-02-19T20:11:21.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:21.243+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:21.244710+0000) 2024-02-19T20:11:21.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:21.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:21.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:21.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:21.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:21.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:21.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:21.245+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:21.925 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:21.588+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56076 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:21.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:21.585+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:21.586707+0000) 2024-02-19T20:11:21.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:21.767+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:21.768218+0000) 2024-02-19T20:11:21.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:21.585+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:21.586896+0000) 2024-02-19T20:11:21.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:21.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:21.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:21.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:21.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:21.699+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:21.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:21.700+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:21.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:21.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:21.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:21.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:21.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:21.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:21.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:21.577+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:21.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:21.585+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:21.586806+0000) 2024-02-19T20:11:22.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:21.985+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:21.986960+0000) 2024-02-19T20:11:22.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:22.243+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:22.244882+0000) 2024-02-19T20:11:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:22.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:22.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:22.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:22.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:22.228+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:22.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:22.769+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:22.770194+0000) 2024-02-19T20:11:22.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:22.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:22.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:22.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:22.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:22.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:22.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:22.603+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:22.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:22.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:22.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:22.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:22.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:22.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:22.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:22.722+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:23.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:22.987+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:22.988993+0000) 2024-02-19T20:11:23.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:23.246+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:23.247010+0000) 2024-02-19T20:11:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:23.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:23.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:23.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:23.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:23.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:23.772+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:23.773727+0000) 2024-02-19T20:11:23.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:23.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:23.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:23.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:23.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:23.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:23.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:23.585+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:23.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:23.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:23.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:23.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:23.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:23.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:23.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:23.751+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:24.249 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:23.990+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:23.991052+0000) 2024-02-19T20:11:24.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:24.248+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:24.249088+0000) 2024-02-19T20:11:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:24.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:24.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:24.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:24.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:24.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:24.307+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:24.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:24.774+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:24.775503+0000) 2024-02-19T20:11:24.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:24.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:24.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:24.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:24.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:24.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:24.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:24.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:24.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:24.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:24.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:24.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:24.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:24.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:24.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:24.795+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:25.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:24.991+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:24.992938+0000) 2024-02-19T20:11:25.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:25.250+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:25.251083+0000) 2024-02-19T20:11:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:25.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:25.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:25.287+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:25.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:25.287+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:25.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:25.774+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:25.775678+0000) 2024-02-19T20:11:25.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:25.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:25.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:25.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:25.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:25.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:25.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:25.801+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:25.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:25.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:25.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:25.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:25.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:25.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:25.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:25.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:26.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:25.992+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:25.993126+0000) 2024-02-19T20:11:26.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:26.250+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:26.251282+0000) 2024-02-19T20:11:26.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:26.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:26.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:26.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:26.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:26.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:26.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:26.297+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:26.926 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:26.595+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56115 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:26.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:26.593+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:26.594384+0000) 2024-02-19T20:11:26.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:26.774+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:26.775888+0000) 2024-02-19T20:11:26.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:26.592+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:26.593820+0000) 2024-02-19T20:11:26.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:26.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:26.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:26.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:26.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:26.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:26.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:26.795+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:26.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:26.593+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:26.594823+0000) 2024-02-19T20:11:26.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:26.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:26.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:26.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:26.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:26.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:26.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:26.681+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:27.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:26.992+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:26.993275+0000) 2024-02-19T20:11:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:27.250+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:27.251446+0000) 2024-02-19T20:11:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:27.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:27.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:27.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:27.262+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:27.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:27.777+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:27.778472+0000) 2024-02-19T20:11:27.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:27.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:27.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:27.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:27.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:27.693+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:27.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:27.693+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:27.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:27.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:27.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:27.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:27.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:27.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:27.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:27.832+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:28.240 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:27.994+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:27.995387+0000) 2024-02-19T20:11:28.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:28.252+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:28.253668+0000) 2024-02-19T20:11:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:28.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:28.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:28.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:28.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:28.239+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:28.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:28.779+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:28.780682+0000) 2024-02-19T20:11:28.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:28.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:28.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:28.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:28.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:28.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:28.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:28.832+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:28.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:28.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:28.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:28.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:28.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:28.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:28.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:28.726+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:29.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:28.996+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:28.997474+0000) 2024-02-19T20:11:29.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:29.255+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:29.256125+0000) 2024-02-19T20:11:29.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:29.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:29.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:29.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:29.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:29.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:29.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:29.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:29.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:29.781+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:29.782495+0000) 2024-02-19T20:11:29.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:29.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:29.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:29.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:29.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:29.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:29.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:29.715+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:29.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:29.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:29.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:29.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:29.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:29.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:29.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:29.823+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:30.189 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:29.998+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:29.999352+0000) 2024-02-19T20:11:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:30.256+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:30.257993+0000) 2024-02-19T20:11:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:30.231+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:30.231+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:30.231+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:30.232+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:30.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:30.781+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:30.782695+0000) 2024-02-19T20:11:30.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:30.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:30.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:30.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:30.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:30.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:30.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:30.671+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:30.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:30.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:30.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:30.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:30.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:30.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:30.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:30.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:31.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:30.998+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:30.999524+0000) 2024-02-19T20:11:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:31.257+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:31.258195+0000) 2024-02-19T20:11:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:31.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:31.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:31.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:31.248+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:31.891 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:31.600+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:31.601364+0000) 2024-02-19T20:11:31.891 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:31.781+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:31.782883+0000) 2024-02-19T20:11:31.891 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:31.599+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:31.600871+0000) 2024-02-19T20:11:31.892 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:31.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:31.892 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:31.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:31.892 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:31.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:31.892 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:31.693+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:31.892 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:31.603+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56154 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:31.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:31.599+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:31.600935+0000) 2024-02-19T20:11:31.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:31.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:31.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:31.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:31.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:31.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:32.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:31.890+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:32.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:31.998+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:31.999707+0000) 2024-02-19T20:11:32.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:32.257+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:32.258348+0000) 2024-02-19T20:11:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:32.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:32.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:32.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:32.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:32.210+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:32.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:32.784+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:32.785411+0000) 2024-02-19T20:11:32.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:32.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:32.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:32.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:32.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:32.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:32.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:32.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:33.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:32.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:33.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:32.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:33.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:32.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:33.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:32.897+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:33.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:33.002+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:33.003119+0000) 2024-02-19T20:11:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:33.259+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:33.260281+0000) 2024-02-19T20:11:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:33.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:33.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:33.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:33.233+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:33.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:33.786+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:33.787427+0000) 2024-02-19T20:11:33.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:33.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:33.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:33.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:33.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:33.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:33.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:33.677+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:33.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:33.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:33.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:33.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:33.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:33.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:33.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:33.865+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:34.189 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:34.005+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:34.006053+0000) 2024-02-19T20:11:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:34.261+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:34.262250+0000) 2024-02-19T20:11:34.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:34.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:34.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:34.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:34.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:34.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:34.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:34.189+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:34.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:34.788+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:34.789214+0000) 2024-02-19T20:11:34.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:34.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:34.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:34.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:34.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:34.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:34.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:34.642+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:34.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:34.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:34.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:34.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:34.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:34.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:34.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:34.843+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:35.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:35.006+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:35.007852+0000) 2024-02-19T20:11:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:35.263+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:35.264034+0000) 2024-02-19T20:11:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:35.233+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:35.233+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:35.233+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:35.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:35.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:35.788+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:35.789474+0000) 2024-02-19T20:11:35.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:35.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:35.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:35.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:35.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:35.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:35.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:35.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:35.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:35.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:35.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:35.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:35.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:35.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:35.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:35.864+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:36.214 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:36.007+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:36.008069+0000) 2024-02-19T20:11:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:36.263+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:36.264185+0000) 2024-02-19T20:11:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:36.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:36.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:36.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:36.213+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:36.927 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:36.610+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56193 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:36.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:36.606+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:36.607890+0000) 2024-02-19T20:11:36.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:36.788+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:36.789674+0000) 2024-02-19T20:11:36.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:36.607+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:36.608763+0000) 2024-02-19T20:11:36.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:36.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:36.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:36.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:36.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:36.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:36.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:36.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:36.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:36.607+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:36.608249+0000) 2024-02-19T20:11:36.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:36.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:36.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:36.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:36.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:36.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:36.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:36.837+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:37.189 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:37.007+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:37.008218+0000) 2024-02-19T20:11:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:37.263+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:37.264336+0000) 2024-02-19T20:11:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:37.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:37.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:37.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:37.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:37.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:37.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:37.792+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:37.793284+0000) 2024-02-19T20:11:37.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:37.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:37.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:37.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:37.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:37.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:37.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:37.856+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:37.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:37.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:37.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:37.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:37.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:37.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:37.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:37.609+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:38.267 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:38.009+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:38.010594+0000) 2024-02-19T20:11:38.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:38.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:38.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:38.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:38.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:38.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:38.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:38.148+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:38.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:38.265+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:38.266683+0000) 2024-02-19T20:11:38.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:38.795+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:38.796895+0000) 2024-02-19T20:11:38.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:38.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:38.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:38.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:38.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:38.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:38.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:38.586+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:38.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:38.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:38.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:38.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:38.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:38.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:39.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:38.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:39.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:39.012+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.013212+0000) 2024-02-19T20:11:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:39.269+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.270321+0000) 2024-02-19T20:11:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:39.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:39.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:39.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:39.151+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:39.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:39.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:39.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:39.631+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:40.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:39.797+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:39.798720+0000) 2024-02-19T20:11:40.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:39.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:40.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:39.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:40.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:39.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:40.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:39.873+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:40.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:40.014+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:40.015452+0000) 2024-02-19T20:11:40.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:40.272+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:40.273482+0000) 2024-02-19T20:11:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:40.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:40.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:40.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:40.156+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:40.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:40.797+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:40.798968+0000) 2024-02-19T20:11:40.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:40.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:40.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:40.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:40.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:40.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:40.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:40.614+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:40.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:40.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:40.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:40.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:40.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:40.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:40.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:40.851+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:41.179 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:41.014+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.015653+0000) 2024-02-19T20:11:41.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:41.272+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.273659+0000) 2024-02-19T20:11:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:41.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:41.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:41.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:41.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:41.178+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:41.876 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:41.617+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56232 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:41.876 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:41.615+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.616207+0000) 2024-02-19T20:11:41.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:41.614+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.615275+0000) 2024-02-19T20:11:41.877 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:41.798+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.799263+0000) 2024-02-19T20:11:41.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:41.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:41.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:41.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:41.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:41.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:41.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:41.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:41.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:41.614+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:41.615735+0000) 2024-02-19T20:11:42.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:41.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:42.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:41.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:42.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:41.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:42.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:41.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:42.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:42.014+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:42.015840+0000) 2024-02-19T20:11:42.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:42.272+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:42.273864+0000) 2024-02-19T20:11:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:42.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:42.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:42.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:42.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:42.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:42.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:42.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:42.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:42.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:42.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:42.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:42.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:43.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:42.800+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:42.801291+0000) 2024-02-19T20:11:43.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:42.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:43.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:42.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:43.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:42.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:43.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:42.916+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:43.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:43.016+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:43.017896+0000) 2024-02-19T20:11:43.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:43.275+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:43.276058+0000) 2024-02-19T20:11:43.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:43.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:43.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:43.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:43.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:43.174+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:43.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:43.174+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:43.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:43.802+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:43.803683+0000) 2024-02-19T20:11:43.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:43.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:43.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:43.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:43.928 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:43.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:43.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:43.571+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:44.187 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:43.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:44.187 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:43.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:44.187 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:43.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:44.187 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:43.938+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:44.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:44.020+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.021350+0000) 2024-02-19T20:11:44.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:44.277+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.278057+0000) 2024-02-19T20:11:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:44.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:44.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:44.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:44.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:44.186+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:44.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:44.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:44.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:44.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:44.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:44.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:44.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:44.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:45.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:44.804+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:44.805483+0000) 2024-02-19T20:11:45.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:44.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:45.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:44.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:45.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:44.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:45.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:44.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:45.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:45.023+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:45.023164+0000) 2024-02-19T20:11:45.544 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:45.280+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:45.280306+0000) 2024-02-19T20:11:45.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:45.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:45.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:45.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:45.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:45.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:45.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:45.191+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:45.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:45.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:45.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:45.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:45.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:45.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:45.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:45.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:46.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:45.805+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:45.805788+0000) 2024-02-19T20:11:46.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:45.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:46.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:45.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:46.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:45.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:46.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:45.938+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:46.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:46.023+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.023331+0000) 2024-02-19T20:11:46.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:46.280+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.280544+0000) 2024-02-19T20:11:46.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:46.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:46.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:46.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:46.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:46.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:46.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:46.225+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:46.926 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:46.625+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56271 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:46.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:46.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:46.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:46.584+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:46.622+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.622281+0000) 2024-02-19T20:11:46.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:46.623+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.623111+0000) 2024-02-19T20:11:46.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:46.806+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.806081+0000) 2024-02-19T20:11:46.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:46.622+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:46.622806+0000) 2024-02-19T20:11:46.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:46.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:46.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:46.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:46.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:46.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:47.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:46.926+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:47.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:47.023+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:47.023528+0000) 2024-02-19T20:11:47.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:47.279+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:47.280737+0000) 2024-02-19T20:11:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:47.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:47.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:47.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:47.187+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:47.888 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:47.807+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:47.808099+0000) 2024-02-19T20:11:47.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:47.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:47.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:47.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:47.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:47.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:47.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:47.599+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:47.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:47.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:47.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:47.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:47.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:47.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:48.186 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:47.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:48.186 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:48.024+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:48.025673+0000) 2024-02-19T20:11:48.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:48.283+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:48.284266+0000) 2024-02-19T20:11:48.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:48.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:48.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:48.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:48.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:48.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:48.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:48.185+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:48.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:48.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:48.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:48.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:48.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:48.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:48.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:48.553+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:49.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:48.809+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:48.810152+0000) 2024-02-19T20:11:49.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:48.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:49.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:48.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:49.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:48.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:49.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:48.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:49.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:49.026+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.027720+0000) 2024-02-19T20:11:49.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:49.285+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.286330+0000) 2024-02-19T20:11:49.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:49.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:49.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:49.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:49.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:49.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:49.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:49.212+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:49.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:49.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:49.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:49.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:49.540+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:50.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:49.811+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:49.812080+0000) 2024-02-19T20:11:50.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:49.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:50.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:49.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:50.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:49.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:50.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:49.860+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:50.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:50.028+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:50.029489+0000) 2024-02-19T20:11:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:50.287+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:50.288180+0000) 2024-02-19T20:11:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:50.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:50.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:50.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:50.252+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:50.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:50.811+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:50.812248+0000) 2024-02-19T20:11:50.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:50.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:50.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:50.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:50.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:50.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:50.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:50.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:50.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:50.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:50.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:50.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:50.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:50.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:50.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:50.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:51.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:51.028+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.029672+0000) 2024-02-19T20:11:51.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:51.287+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.288368+0000) 2024-02-19T20:11:51.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:51.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:51.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:51.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:51.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:51.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:51.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:51.225+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:51.929 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:51.631+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56310 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:51.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:51.628+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.629815+0000) 2024-02-19T20:11:51.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:51.811+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.812445+0000) 2024-02-19T20:11:51.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:51.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:51.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:51.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:51.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:51.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:51.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:51.571+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:51.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:51.628+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.630001+0000) 2024-02-19T20:11:51.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:51.628+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:51.629329+0000) 2024-02-19T20:11:51.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:51.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:51.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:51.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:51.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:51.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:51.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:51.793+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:52.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:52.028+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:52.029892+0000) 2024-02-19T20:11:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:52.287+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:52.288560+0000) 2024-02-19T20:11:52.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:52.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:52.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:52.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:52.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:52.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:52.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:52.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:52.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:52.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:52.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:52.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:52.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:52.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:52.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:52.546+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:53.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:52.814+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:52.815633+0000) 2024-02-19T20:11:53.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:52.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:53.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:52.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:53.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:52.838+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:53.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:52.838+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:53.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:53.031+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:53.032415+0000) 2024-02-19T20:11:53.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:53.289+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:53.290545+0000) 2024-02-19T20:11:53.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:53.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:53.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:53.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:53.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:53.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:53.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:53.231+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:53.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:53.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:53.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:53.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:53.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:53.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:53.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:53.526+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:53.817+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:53.818362+0000) 2024-02-19T20:11:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:53.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:53.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:53.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:53.790+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:54.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:54.033+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.034684+0000) 2024-02-19T20:11:54.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:54.293+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.294172+0000) 2024-02-19T20:11:54.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:54.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:54.483 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:54.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:54.483 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:54.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:54.483 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:54.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:54.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:54.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:54.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:54.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:54.482+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:55.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:54.819+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:54.820212+0000) 2024-02-19T20:11:55.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:54.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:55.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:54.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:55.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:54.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:55.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:54.792+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:55.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:55.035+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:55.036522+0000) 2024-02-19T20:11:55.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:55.296+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:55.297331+0000) 2024-02-19T20:11:55.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:55.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:55.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:55.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:55.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:55.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:55.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:55.239+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:55.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:55.239+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:55.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:55.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:55.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:55.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:55.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:55.483+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:56.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:55.819+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:55.820392+0000) 2024-02-19T20:11:56.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:55.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:56.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:55.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:56.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:55.762+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:56.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:55.763+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:56.035+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.036692+0000) 2024-02-19T20:11:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:56.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:56.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:56.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:56.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:56.222+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:56.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:56.296+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.297538+0000) 2024-02-19T20:11:56.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:56.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:56.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:56.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:56.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:56.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:56.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:56.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:56.929 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:11:56.638+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56349 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:11:56.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:56.635+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.636604+0000) 2024-02-19T20:11:56.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:56.635+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.636053+0000) 2024-02-19T20:11:56.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:56.819+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.820606+0000) 2024-02-19T20:11:56.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:56.635+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:56.636321+0000) 2024-02-19T20:11:56.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:56.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:56.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:56.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:56.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:56.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:56.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:56.736+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:57.224 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:57.035+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:57.036902+0000) 2024-02-19T20:11:57.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:57.296+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:57.297803+0000) 2024-02-19T20:11:57.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:57.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:57.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:57.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:57.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:57.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:57.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:57.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:57.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:57.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:57.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:57.224+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:57.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:57.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:57.495+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:57.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:57.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:57.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:57.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:57.707+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:58.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:58.037+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:58.038915+0000) 2024-02-19T20:11:58.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:57.821+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:57.822634+0000) 2024-02-19T20:11:58.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:58.298+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:58.299816+0000) 2024-02-19T20:11:58.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:58.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:58.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:58.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:58.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:58.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:58.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:58.242+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:58.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:58.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:58.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:58.674+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:11:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:58.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:58.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:58.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:58.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:58.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:59.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:58.824+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:58.825212+0000) 2024-02-19T20:11:59.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:59.040+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.041387+0000) 2024-02-19T20:11:59.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:59.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:59.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:59.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:59.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:59.204+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:11:59.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:59.204+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:59.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:59.300+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.301850+0000) 2024-02-19T20:11:59.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:59.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:59.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:59.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:59.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:59.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:11:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:11:59.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:11:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:59.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:59.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:59.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:11:59.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:11:59.723+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:00.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:00.042+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:00.043272+0000) 2024-02-19T20:12:00.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:11:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:11:59.826+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:11:59.827056+0000) 2024-02-19T20:12:00.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:00.303+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:00.304109+0000) 2024-02-19T20:12:00.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:00.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:00.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:00.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:00.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:00.524+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:00.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:00.216+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:00.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:00.216+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:00.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:00.216+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:00.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:00.216+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:00.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:00.524+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:00.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:00.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:00.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:00.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:00.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:00.740+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:01.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:01.042+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.043458+0000) 2024-02-19T20:12:01.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:00.826+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:00.827256+0000) 2024-02-19T20:12:01.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:01.303+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.304288+0000) 2024-02-19T20:12:01.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:01.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:01.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:01.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:01.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:01.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:01.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:01.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:01.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:01.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:01.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:01.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:01.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:01.192+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:01.764 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:01.645+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56388 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:01.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:01.642+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.643428+0000) 2024-02-19T20:12:01.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:01.513+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:01.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:01.641+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.642976+0000) 2024-02-19T20:12:01.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:01.642+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.643618+0000) 2024-02-19T20:12:01.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:01.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:01.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:01.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:01.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:01.721+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:01.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:01.722+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:01.826+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:01.827428+0000) 2024-02-19T20:12:02.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:02.042+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:02.043671+0000) 2024-02-19T20:12:02.305 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:02.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:02.305 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:02.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:02.305 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:02.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:02.305 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:02.236+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:02.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:02.303+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:02.304453+0000) 2024-02-19T20:12:02.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:02.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:02.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:02.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:02.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:02.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:02.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:02.519+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:02.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:02.829+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:02.830815+0000) 2024-02-19T20:12:02.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:02.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:02.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:02.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:02.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:02.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:02.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:02.764+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:03.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:03.044+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:03.045715+0000) 2024-02-19T20:12:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:03.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:03.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:03.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:03.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:03.252+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:03.305+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:03.306591+0000) 2024-02-19T20:12:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:03.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:03.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:03.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:03.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:03.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:03.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:03.833+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:03.834164+0000) 2024-02-19T20:12:03.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:03.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:03.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:03.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:03.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:03.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:03.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:03.751+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:04.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:04.046+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.047887+0000) 2024-02-19T20:12:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:04.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:04.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:04.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:04.271+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:04.307+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.308604+0000) 2024-02-19T20:12:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:04.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:04.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:04.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:04.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:04.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:04.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:04.836+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:04.837313+0000) 2024-02-19T20:12:04.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:04.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:04.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:04.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:04.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:04.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:04.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:04.775+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:05.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:05.048+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:05.049767+0000) 2024-02-19T20:12:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:05.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:05.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:05.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:05.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:05.274+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:05.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:05.310+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:05.311051+0000) 2024-02-19T20:12:05.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:05.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:05.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:05.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:05.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:05.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:05.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:05.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:05.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:05.836+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:05.837514+0000) 2024-02-19T20:12:05.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:05.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:05.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:05.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:05.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:05.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:05.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:05.773+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:06.048+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.049960+0000) 2024-02-19T20:12:06.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:06.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:06.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:06.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:06.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:06.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:06.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:06.298+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:06.648 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:06.310+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.311205+0000) 2024-02-19T20:12:06.648 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:06.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:06.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:06.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:06.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:06.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:06.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:06.459+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:06.930 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:06.653+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56427 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:06.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:06.649+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.650423+0000) 2024-02-19T20:12:06.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:06.649+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.650907+0000) 2024-02-19T20:12:06.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:06.836+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.837710+0000) 2024-02-19T20:12:06.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:06.649+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:06.650811+0000) 2024-02-19T20:12:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:06.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:06.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:06.791+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:06.792+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:07.049+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:07.050138+0000) 2024-02-19T20:12:07.655 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:07.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:07.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:07.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:07.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:07.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:07.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:07.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:07.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:07.310+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:07.311358+0000) 2024-02-19T20:12:07.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:07.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:07.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:07.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:07.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:07.480+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:07.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:07.481+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:07.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:07.838+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:07.839712+0000) 2024-02-19T20:12:07.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:07.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:07.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:07.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:07.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:07.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:07.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:07.779+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:08.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:08.051+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:08.052116+0000) 2024-02-19T20:12:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:08.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:08.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:08.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:08.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:08.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:08.312+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:08.313831+0000) 2024-02-19T20:12:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:08.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:08.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:08.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:08.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:08.531+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:08.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:08.840+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:08.841951+0000) 2024-02-19T20:12:08.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:08.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:08.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:08.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:08.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:08.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:08.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:08.772+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:09.054+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.055595+0000) 2024-02-19T20:12:09.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:09.314+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.315930+0000) 2024-02-19T20:12:09.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:09.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:09.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:09.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:09.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:09.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:09.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:09.526+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:09.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:09.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:09.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:09.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:09.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:09.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:09.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:09.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:09.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:09.842+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:09.843811+0000) 2024-02-19T20:12:09.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:09.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:09.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:09.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:09.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:09.803+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:09.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:09.804+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:10.191 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:10.057+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:10.058754+0000) 2024-02-19T20:12:10.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:10.316+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:10.318000+0000) 2024-02-19T20:12:10.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:10.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:10.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:10.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:10.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:10.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:10.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:10.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:10.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:10.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:10.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:10.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:10.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:10.336+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:10.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:10.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:11.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:10.842+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:10.843990+0000) 2024-02-19T20:12:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:10.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:10.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:10.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:11.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:10.780+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:11.317 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:11.057+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.058946+0000) 2024-02-19T20:12:11.655 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:11.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:11.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:11.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:11.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:11.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:11.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:11.364+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:11.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:11.317+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.318185+0000) 2024-02-19T20:12:11.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:11.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:11.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:11.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:11.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:11.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:11.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:11.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:11.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:11.657+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.658503+0000) 2024-02-19T20:12:11.931 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:11.660+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56465 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:11.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:11.658+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.659486+0000) 2024-02-19T20:12:11.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:11.843+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.844170+0000) 2024-02-19T20:12:11.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:11.657+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:11.658639+0000) 2024-02-19T20:12:11.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:11.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:11.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:11.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:11.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:11.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:11.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:11.783+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:12.058+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:12.059160+0000) 2024-02-19T20:12:12.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:12.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:12.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:12.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:12.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:12.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:12.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:12.353+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:12.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:12.317+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:12.318348+0000) 2024-02-19T20:12:12.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:12.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:12.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:12.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:12.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:12.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:12.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:12.506+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:12.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:12.845+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:12.846188+0000) 2024-02-19T20:12:12.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:12.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:12.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:12.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:12.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:12.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:12.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:12.801+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:13.060+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:13.061269+0000) 2024-02-19T20:12:13.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:13.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:13.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:13.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:13.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:13.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:13.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:13.400+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:13.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:13.319+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:13.320314+0000) 2024-02-19T20:12:13.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:13.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:13.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:13.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:13.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:13.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:13.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:13.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:13.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:13.847+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:13.848200+0000) 2024-02-19T20:12:13.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:13.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:13.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:13.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:13.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:13.756+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:13.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:13.757+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:14.062+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.063305+0000) 2024-02-19T20:12:14.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:14.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:14.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:14.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:14.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:14.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:14.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:14.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:14.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:14.322+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.323910+0000) 2024-02-19T20:12:14.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:14.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:14.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:14.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:14.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:14.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:14.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:14.546+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:14.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:14.849+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:14.850280+0000) 2024-02-19T20:12:14.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:14.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:14.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:14.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:14.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:14.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:14.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:14.790+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:15.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:15.064+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:15.065124+0000) 2024-02-19T20:12:15.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:15.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:15.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:15.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:15.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:15.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:15.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:15.403+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:15.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:15.324+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:15.325774+0000) 2024-02-19T20:12:15.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:15.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:15.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:15.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:15.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:15.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:15.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:15.558+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:15.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:15.849+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:15.850475+0000) 2024-02-19T20:12:15.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:15.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:15.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:15.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:15.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:15.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:15.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:15.752+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:16.064+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.065285+0000) 2024-02-19T20:12:16.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:16.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:16.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:16.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:16.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:16.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:16.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:16.425+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:16.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:16.324+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.325976+0000) 2024-02-19T20:12:16.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:16.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:16.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:16.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:16.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:16.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:16.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:16.559+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:16.931 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:16.667+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56496 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:16.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:16.664+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.665262+0000) 2024-02-19T20:12:16.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:16.664+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.665286+0000) 2024-02-19T20:12:16.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:16.849+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.850727+0000) 2024-02-19T20:12:16.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:16.664+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:16.665494+0000) 2024-02-19T20:12:16.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:16.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:16.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:16.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:16.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:16.794+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:16.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:16.795+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:17.064+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:17.065454+0000) 2024-02-19T20:12:17.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:17.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:17.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:17.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:17.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:17.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:17.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:17.453+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:17.325+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:17.326153+0000) 2024-02-19T20:12:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:17.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:17.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:17.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:17.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:17.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:17.852+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:17.853398+0000) 2024-02-19T20:12:17.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:17.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:17.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:17.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:17.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:17.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:17.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:17.755+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:18.067+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:18.068112+0000) 2024-02-19T20:12:18.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:18.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:18.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:18.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:18.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:18.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:18.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:18.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:18.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:18.329+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:18.330765+0000) 2024-02-19T20:12:18.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:18.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:18.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:18.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:18.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:18.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:18.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:18.554+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:18.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:18.855+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:18.856062+0000) 2024-02-19T20:12:18.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:18.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:18.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:18.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:18.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:18.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:18.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:18.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:19.069+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.070137+0000) 2024-02-19T20:12:19.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:19.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:19.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:19.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:19.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:19.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:19.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:19.466+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:19.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:19.333+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.334454+0000) 2024-02-19T20:12:19.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:19.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:19.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:19.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:19.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:19.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:19.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:19.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:19.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:19.857+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:19.858017+0000) 2024-02-19T20:12:19.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:19.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:19.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:19.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:19.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:19.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:19.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:19.751+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:20.192 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:20.071+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:20.072176+0000) 2024-02-19T20:12:20.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:20.335+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:20.336764+0000) 2024-02-19T20:12:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:20.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:20.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:20.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:20.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:20.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:20.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:20.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:20.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:20.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:20.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:20.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:20.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:20.572+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:20.859 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:20.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:20.859 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:20.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:20.859 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:20.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:20.859 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:20.738+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:21.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:21.071+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.072345+0000) 2024-02-19T20:12:21.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:20.857+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:20.858198+0000) 2024-02-19T20:12:21.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:21.336+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.337016+0000) 2024-02-19T20:12:21.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:21.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:21.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:21.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:21.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:21.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:21.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:21.418+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:21.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:21.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:21.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:21.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:21.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:21.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:21.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:21.585+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:21.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:21.671+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.672579+0000) 2024-02-19T20:12:21.932 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:21.674+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56536 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:21.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:21.671+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.672522+0000) 2024-02-19T20:12:21.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:21.857+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.858364+0000) 2024-02-19T20:12:21.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:21.672+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:21.673142+0000) 2024-02-19T20:12:21.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:21.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:21.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:21.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:21.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:21.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:21.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:21.725+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:22.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:22.071+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:22.072626+0000) 2024-02-19T20:12:22.588 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:22.336+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:22.337285+0000) 2024-02-19T20:12:22.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:22.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:22.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:22.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:22.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:22.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:22.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:22.387+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:22.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:22.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:22.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:22.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:22.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:22.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:22.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:22.587+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:22.861 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:22.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:22.861 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:22.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:22.861 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:22.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:22.861 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:22.689+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:23.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:23.075+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:23.076149+0000) 2024-02-19T20:12:23.135 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:22.859+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:22.860347+0000) 2024-02-19T20:12:23.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:23.338+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:23.339447+0000) 2024-02-19T20:12:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:23.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:23.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:23.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:23.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:23.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:23.357+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:23.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:23.861+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:23.862586+0000) 2024-02-19T20:12:23.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:23.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:23.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:23.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:23.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:23.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:23.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:23.631+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:23.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:23.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:23.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:23.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:23.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:23.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:23.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:23.734+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:24.077+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.078142+0000) 2024-02-19T20:12:24.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:24.340+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.341462+0000) 2024-02-19T20:12:24.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:24.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:24.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:24.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:24.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:24.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:24.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:24.335+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:24.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:24.863+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:24.864610+0000) 2024-02-19T20:12:24.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:24.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:24.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:24.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:24.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:24.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:24.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:24.680+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:24.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:24.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:24.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:24.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:24.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:24.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:24.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:24.717+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:25.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:25.080+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:25.081486+0000) 2024-02-19T20:12:25.646 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:25.343+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:25.344853+0000) 2024-02-19T20:12:25.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:25.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:25.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:25.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:25.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:25.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:25.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:25.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:25.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:25.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:25.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:25.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:25.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:25.322+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:25.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:25.646+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:25.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:25.863+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:25.864812+0000) 2024-02-19T20:12:25.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:25.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:25.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:25.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:25.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:25.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:25.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:25.761+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:26.080+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.081663+0000) 2024-02-19T20:12:26.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:26.344+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.345009+0000) 2024-02-19T20:12:26.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:26.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:26.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:26.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:26.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:26.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:26.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:26.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:26.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:26.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:26.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:26.299+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:26.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:26.299+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:26.932 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:26.681+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56574 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:26.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:26.654+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:26.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:26.678+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.679566+0000) 2024-02-19T20:12:26.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:26.678+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.679802+0000) 2024-02-19T20:12:26.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:26.863+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.865001+0000) 2024-02-19T20:12:26.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:26.678+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:26.679588+0000) 2024-02-19T20:12:26.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:26.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:26.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:26.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:26.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:26.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:26.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:26.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:27.080+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:27.081862+0000) 2024-02-19T20:12:27.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:27.344+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:27.345201+0000) 2024-02-19T20:12:27.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:27.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:27.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:27.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:27.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:27.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:27.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:27.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:27.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:27.867+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:27.868642+0000) 2024-02-19T20:12:27.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:27.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:27.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:27.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:27.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:27.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:27.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:27.735+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:27.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:27.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:27.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:27.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:27.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:27.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:27.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:27.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:28.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:28.084+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:28.085394+0000) 2024-02-19T20:12:28.629 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:28.346+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:28.347069+0000) 2024-02-19T20:12:28.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:28.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:28.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:28.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:28.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:28.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:28.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:28.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:28.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:28.869+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:28.870646+0000) 2024-02-19T20:12:28.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:28.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:28.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:28.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:28.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:28.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:28.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:28.629+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:28.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:28.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:28.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:28.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:28.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:28.745+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:28.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:28.746+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:29.087+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.088055+0000) 2024-02-19T20:12:29.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:29.346+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:29.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:29.346+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:29.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:29.346+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:29.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:29.346+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:29.607 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:29.348+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.349044+0000) 2024-02-19T20:12:29.607 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:29.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:29.607 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:29.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:29.607 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:29.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:29.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:29.606+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:29.873 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:29.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:29.873 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:29.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:29.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:29.719+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:29.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:29.720+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:30.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:30.089+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:30.090343+0000) 2024-02-19T20:12:30.135 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:29.872+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:29.873090+0000) 2024-02-19T20:12:30.386 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:30.350+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:30.351473+0000) 2024-02-19T20:12:30.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:30.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:30.654 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:30.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:30.654 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:30.385+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:30.654 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:30.386+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:30.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:30.872+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:30.873260+0000) 2024-02-19T20:12:30.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:30.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:30.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:30.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:30.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:30.652+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:30.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:30.653+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:30.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:30.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:30.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:30.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:30.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:30.757+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:30.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:30.758+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:31.089+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.090547+0000) 2024-02-19T20:12:31.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:31.350+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.351719+0000) 2024-02-19T20:12:31.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:31.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:31.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:31.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:31.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:31.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:31.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:31.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:31.934 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:31.688+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56615 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:31.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:31.686+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.687575+0000) 2024-02-19T20:12:31.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:31.873+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.873471+0000) 2024-02-19T20:12:31.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:31.685+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.686935+0000) 2024-02-19T20:12:31.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:31.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:31.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:31.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:31.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:31.768+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:31.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:31.768+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:31.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:31.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:31.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:31.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:31.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:31.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:31.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:31.670+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:31.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:31.686+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:31.687660+0000) 2024-02-19T20:12:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:32.090+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:32.090720+0000) 2024-02-19T20:12:32.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:32.351+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:32.351937+0000) 2024-02-19T20:12:32.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:32.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:32.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:32.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:32.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:32.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:32.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:32.415+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:32.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:32.877+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:32.877245+0000) 2024-02-19T20:12:32.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:32.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:32.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:32.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:32.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:32.687+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:32.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:32.688+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:32.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:32.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:32.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:32.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:32.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:32.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:32.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:32.759+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:33.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:33.093+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:33.093493+0000) 2024-02-19T20:12:33.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:33.354+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:33.354101+0000) 2024-02-19T20:12:33.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:33.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:33.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:33.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:33.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:33.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:33.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:33.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:33.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:33.880+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:33.880912+0000) 2024-02-19T20:12:33.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:33.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:33.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:33.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:33.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:33.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:33.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:33.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:33.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:33.755+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:33.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:33.755+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:33.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:33.755+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:33.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:33.756+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:34.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:34.094+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.095519+0000) 2024-02-19T20:12:34.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:34.355+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.356467+0000) 2024-02-19T20:12:34.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:34.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:34.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:34.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:34.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:34.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:34.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:34.331+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:34.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:34.881+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:34.882959+0000) 2024-02-19T20:12:34.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:34.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:34.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:34.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:34.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:34.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:34.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:34.755+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:34.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:34.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:34.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:34.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:34.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:34.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:34.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:34.755+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:35.096+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:35.097459+0000) 2024-02-19T20:12:35.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:35.357+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:35.358536+0000) 2024-02-19T20:12:35.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:35.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:35.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:35.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:35.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:35.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:35.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:35.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:35.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:35.882+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:35.883264+0000) 2024-02-19T20:12:35.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:35.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:35.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:35.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:35.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:35.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:35.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:35.788+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:35.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:35.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:35.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:35.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:35.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:35.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:35.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:35.761+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:36.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:36.096+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.097662+0000) 2024-02-19T20:12:36.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:36.357+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.358834+0000) 2024-02-19T20:12:36.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:36.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:36.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:36.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:36.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:36.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:36.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:36.295+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:36.934 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:36.695+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56653 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:36.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:36.693+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.694290+0000) 2024-02-19T20:12:36.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:36.882+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.883461+0000) 2024-02-19T20:12:36.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:36.692+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.693165+0000) 2024-02-19T20:12:36.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:36.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:36.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:36.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:36.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:36.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:36.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:36.824+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:36.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:36.692+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:36.693697+0000) 2024-02-19T20:12:36.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:36.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:36.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:36.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:36.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:36.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:36.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:36.713+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:37.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:37.096+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:37.097812+0000) 2024-02-19T20:12:37.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:37.358+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:37.359061+0000) 2024-02-19T20:12:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:37.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:37.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:37.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:37.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:37.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:37.884+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:37.885675+0000) 2024-02-19T20:12:37.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:37.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:37.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:37.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:37.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:37.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:37.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:37.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:37.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:37.665+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:37.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:37.665+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:37.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:37.665+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:37.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:37.666+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:38.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:38.100+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:38.101185+0000) 2024-02-19T20:12:38.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:38.360+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:38.361061+0000) 2024-02-19T20:12:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:38.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:38.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:38.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:38.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:38.290+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:38.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:38.887+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:38.888280+0000) 2024-02-19T20:12:38.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:38.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:38.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:38.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:38.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:38.808+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:38.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:38.809+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:38.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:38.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:38.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:38.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:38.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:38.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:38.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:38.698+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:39.103+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.104753+0000) 2024-02-19T20:12:39.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:39.362+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.363065+0000) 2024-02-19T20:12:39.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:39.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:39.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:39.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:39.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:39.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:39.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:39.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:39.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:39.889+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:39.890211+0000) 2024-02-19T20:12:39.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:39.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:39.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:39.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:39.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:39.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:39.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:39.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:39.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:39.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:39.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:39.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:39.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:39.688+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:39.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:39.688+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:40.193 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:40.105+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:40.106678+0000) 2024-02-19T20:12:40.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:40.364+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:40.365065+0000) 2024-02-19T20:12:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:40.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:40.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:40.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:40.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:40.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:40.913 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:40.889+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:40.890400+0000) 2024-02-19T20:12:40.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:40.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:40.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:40.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:40.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:40.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:40.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:40.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:40.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:40.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:40.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:40.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:40.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:40.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:40.915 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:40.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:41.105+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.106884+0000) 2024-02-19T20:12:41.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:41.364+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.365239+0000) 2024-02-19T20:12:41.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:41.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:41.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:41.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:41.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:41.359+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:41.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:41.359+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:41.934 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:41.702+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56694 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:41.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:41.699+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.700919+0000) 2024-02-19T20:12:41.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:41.889+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.890664+0000) 2024-02-19T20:12:41.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:41.700+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.701725+0000) 2024-02-19T20:12:41.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:41.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:41.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:41.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:41.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:41.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:41.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:41.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:41.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:41.700+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:41.701298+0000) 2024-02-19T20:12:41.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:41.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:41.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:41.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:41.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:41.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:41.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:41.752+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:42.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:42.106+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:42.107099+0000) 2024-02-19T20:12:42.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:42.364+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:42.365419+0000) 2024-02-19T20:12:42.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:42.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:42.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:42.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:42.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:42.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:42.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:42.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:42.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:42.892+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:42.893100+0000) 2024-02-19T20:12:42.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:42.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:42.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:42.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:42.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:42.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:42.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:42.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:42.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:42.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:42.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:42.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:42.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:42.789+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:42.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:42.790+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:43.108+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:43.109946+0000) 2024-02-19T20:12:43.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:43.366+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:43.367582+0000) 2024-02-19T20:12:43.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:43.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:43.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:43.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:43.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:43.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:43.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:43.423+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:43.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:43.894+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:43.895834+0000) 2024-02-19T20:12:43.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:43.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:43.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:43.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:43.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:43.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:43.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:43.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:43.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:43.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:43.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:43.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:43.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:43.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:43.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:43.820+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:44.110+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.111973+0000) 2024-02-19T20:12:44.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:44.369+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.371009+0000) 2024-02-19T20:12:44.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:44.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:44.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:44.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:44.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:44.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:44.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:44.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:44.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:44.897+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:44.898203+0000) 2024-02-19T20:12:44.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:44.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:44.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:44.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:44.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:44.729+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:44.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:44.730+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:44.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:44.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:44.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:44.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:44.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:44.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:44.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:44.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:45.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:45.112+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:45.113877+0000) 2024-02-19T20:12:45.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:45.372+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:45.373117+0000) 2024-02-19T20:12:45.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:45.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:45.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:45.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:45.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:45.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:45.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:45.432+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:45.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:45.897+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:45.898406+0000) 2024-02-19T20:12:45.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:45.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:45.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:45.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:45.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:45.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:45.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:45.690+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:45.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:45.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:45.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:45.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:45.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:45.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:45.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:45.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:46.113+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.114067+0000) 2024-02-19T20:12:46.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:46.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:46.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:46.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:46.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:46.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:46.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:46.432+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:46.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:46.372+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.373357+0000) 2024-02-19T20:12:46.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:46.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:46.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:46.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:46.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:46.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:46.899 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:46.706+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.707910+0000) 2024-02-19T20:12:46.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:46.710+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56732 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:46.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:46.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:46.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:46.706+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.707782+0000) 2024-02-19T20:12:46.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:46.707+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.709001+0000) 2024-02-19T20:12:46.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:46.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:46.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:46.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:46.901 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:46.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:46.902 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:46.880+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:47.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:47.113+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:47.114253+0000) 2024-02-19T20:12:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:46.897+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:46.898593+0000) 2024-02-19T20:12:47.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:47.372+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:47.373534+0000) 2024-02-19T20:12:47.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:47.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:47.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:47.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:47.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:47.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:47.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:47.445+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:47.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:47.899+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:47.900769+0000) 2024-02-19T20:12:47.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:47.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:47.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:47.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:47.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:47.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:47.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:47.674+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:47.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:47.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:47.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:47.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:47.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:47.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:47.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:47.896+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:48.115+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:48.116281+0000) 2024-02-19T20:12:48.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:48.374+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:48.375563+0000) 2024-02-19T20:12:48.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:48.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:48.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:48.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:48.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:48.411+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:48.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:48.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:48.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:48.901+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:48.902908+0000) 2024-02-19T20:12:48.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:48.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:48.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:48.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:48.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:48.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:48.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:48.724+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:48.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:48.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:48.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:48.924+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:49.117+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.118300+0000) 2024-02-19T20:12:49.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:49.376+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.377614+0000) 2024-02-19T20:12:49.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:49.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:49.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:49.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:49.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:49.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:49.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:49.415+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:49.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:49.903+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:49.904913+0000) 2024-02-19T20:12:49.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:49.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:49.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:49.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:49.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:49.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:49.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:49.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:49.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:49.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:49.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:49.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:49.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:49.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:49.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:49.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:50.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:50.119+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:50.120833+0000) 2024-02-19T20:12:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:50.378+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:50.379722+0000) 2024-02-19T20:12:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:50.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:50.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:50.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:50.369+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:50.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:50.904+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:50.905109+0000) 2024-02-19T20:12:50.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:50.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:50.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:50.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:50.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:50.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:50.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:50.907+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:50.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:50.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:50.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:50.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:50.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:50.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:50.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:50.748+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:51.120+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.121057+0000) 2024-02-19T20:12:51.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:51.378+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.379932+0000) 2024-02-19T20:12:51.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:51.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:51.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:51.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:51.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:51.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:51.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:51.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:51.935 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:51.717+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56773 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:51.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:51.714+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.715450+0000) 2024-02-19T20:12:51.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:51.714+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.715229+0000) 2024-02-19T20:12:51.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:51.904+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.905275+0000) 2024-02-19T20:12:51.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:51.714+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:51.715462+0000) 2024-02-19T20:12:51.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:51.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:51.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:51.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:51.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:51.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:51.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:51.782+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:51.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:51.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:51.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:51.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:52.120+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:52.121232+0000) 2024-02-19T20:12:52.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:52.379+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:52.380118+0000) 2024-02-19T20:12:52.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:52.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:52.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:52.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:52.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:52.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:52.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:52.420+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:52.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:52.906+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:52.907339+0000) 2024-02-19T20:12:52.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:52.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:52.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:52.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:52.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:52.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:52.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:52.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:53.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:52.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:53.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:52.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:53.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:52.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:53.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:52.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:53.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:53.122+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:53.123394+0000) 2024-02-19T20:12:53.468 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:53.381+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:53.382109+0000) 2024-02-19T20:12:53.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:53.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:53.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:53.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:53.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:53.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:53.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:53.468+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:53.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:53.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:53.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:53.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:53.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:53.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:53.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:53.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:53.993 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:53.908+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:53.909473+0000) 2024-02-19T20:12:53.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:53.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:53.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:53.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:53.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:53.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:53.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:53.967+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:54.125+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.126856+0000) 2024-02-19T20:12:54.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:54.383+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.384195+0000) 2024-02-19T20:12:54.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:54.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:54.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:54.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:54.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:54.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:54.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:54.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:54.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:54.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:54.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:54.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:54.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:54.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:54.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:54.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:54.911+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:54.912983+0000) 2024-02-19T20:12:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:55.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.127+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:55.128823+0000) 2024-02-19T20:12:55.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:55.385+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:55.386087+0000) 2024-02-19T20:12:55.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:55.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:55.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:55.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:55.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:55.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:55.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:55.423+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:55.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:55.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:55.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:55.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:55.913 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:55.738+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:55.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:55.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:55.912+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:55.913179+0000) 2024-02-19T20:12:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:55.975+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:56.127+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.128995+0000) 2024-02-19T20:12:56.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:56.385+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.386266+0000) 2024-02-19T20:12:56.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:56.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:56.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:56.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:56.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:56.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:56.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:56.453+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:56.913 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:12:56.723+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56811 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:12:56.914 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:56.720+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.721986+0000) 2024-02-19T20:12:56.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:56.721+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.722471+0000) 2024-02-19T20:12:56.914 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:56.720+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.721900+0000) 2024-02-19T20:12:56.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:56.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:56.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:56.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:56.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:56.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:56.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:56.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:56.912+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:56.913367+0000) 2024-02-19T20:12:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:57.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:57.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:57.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:57.020+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:57.128+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:57.129160+0000) 2024-02-19T20:12:57.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:57.385+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:57.386456+0000) 2024-02-19T20:12:57.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:57.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:57.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:57.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:57.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:57.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:57.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:57.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:57.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:57.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:57.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:57.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:57.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:57.784+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:57.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:57.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:57.915+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:57.916156+0000) 2024-02-19T20:12:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:58.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:58.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:58.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:58.012+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:58.130+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:58.131346+0000) 2024-02-19T20:12:58.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:58.387+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:58.388616+0000) 2024-02-19T20:12:58.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:58.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:58.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:58.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:58.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:58.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:58.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:58.405+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:58.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:58.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:58.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:58.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:58.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:58.814+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:58.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:58.815+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:58.919+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:58.920205+0000) 2024-02-19T20:12:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:59.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:59.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:59.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:12:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:59.055+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:12:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:12:59.132+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.133531+0000) 2024-02-19T20:12:59.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:59.390+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.391147+0000) 2024-02-19T20:12:59.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:59.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:59.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:59.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:59.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:59.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:12:59.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:59.423+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:12:59.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:59.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:59.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:59.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:59.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:59.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:12:59.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:12:59.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:00.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:12:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:12:59.921+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:12:59.922741+0000) 2024-02-19T20:13:00.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:00.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:00.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:00.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:00.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:00.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:00.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:00.035+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:00.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:00.134+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:00.135949+0000) 2024-02-19T20:13:00.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:00.392+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:00.393555+0000) 2024-02-19T20:13:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:00.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:00.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:00.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:00.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:00.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:00.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:00.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:00.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:00.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:00.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:00.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:00.786+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:00.922+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:00.923016+0000) 2024-02-19T20:13:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.135+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.136163+0000) 2024-02-19T20:13:01.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:01.392+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.393809+0000) 2024-02-19T20:13:01.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:01.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:01.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:01.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:01.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:01.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:01.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:01.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:01.923 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:01.730+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56852 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:01.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:01.728+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.729534+0000) 2024-02-19T20:13:01.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:01.728+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.729683+0000) 2024-02-19T20:13:01.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:01.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:01.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:01.757+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:01.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:01.758+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:01.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.727+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.728797+0000) 2024-02-19T20:13:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:01.922+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:01.923218+0000) 2024-02-19T20:13:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:01.994+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:02.135+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:02.136407+0000) 2024-02-19T20:13:02.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:02.393+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:02.394064+0000) 2024-02-19T20:13:02.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:02.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:02.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:02.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:02.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:02.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:02.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:02.425+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:02.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:02.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:02.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:02.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:02.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:02.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:02.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:02.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:02.925+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:02.926240+0000) 2024-02-19T20:13:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:02.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:02.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:02.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:02.955+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:03.137+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:03.138485+0000) 2024-02-19T20:13:03.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:03.395+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:03.396098+0000) 2024-02-19T20:13:03.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:03.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:03.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:03.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:03.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:03.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:03.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:03.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:03.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:03.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:03.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:03.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:03.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:03.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:03.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:03.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:03.927+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:03.928730+0000) 2024-02-19T20:13:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:03.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:03.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:03.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:03.952+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:04.139+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.140953+0000) 2024-02-19T20:13:04.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:04.397+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.398932+0000) 2024-02-19T20:13:04.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:04.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:04.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:04.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:04.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:04.488+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:04.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:04.489+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:04.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:04.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:04.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:04.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:04.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:04.803+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:04.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:04.803+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:04.929+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:04.930864+0000) 2024-02-19T20:13:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:04.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:04.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:04.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:04.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:05.141+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:05.142955+0000) 2024-02-19T20:13:05.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:05.399+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:05.400900+0000) 2024-02-19T20:13:05.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:05.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:05.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:05.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:05.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:05.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:05.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:05.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:05.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:05.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:05.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:05.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:05.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:05.851+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:05.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:05.852+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:05.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:05.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:05.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:05.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:05.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:05.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:05.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:05.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:06.142+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.143151+0000) 2024-02-19T20:13:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:05.930+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:05.931056+0000) 2024-02-19T20:13:06.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:06.400+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.401064+0000) 2024-02-19T20:13:06.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:06.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:06.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:06.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:06.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:06.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:06.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:06.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:06.931 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:06.737+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56890 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:06.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:06.735+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.736464+0000) 2024-02-19T20:13:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:06.735+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.736621+0000) 2024-02-19T20:13:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:06.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:06.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:06.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:06.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:06.872+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:06.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:06.735+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.736142+0000) 2024-02-19T20:13:06.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:06.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:06.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:06.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:06.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:06.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:06.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:06.844+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:07.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:07.142+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:07.143316+0000) 2024-02-19T20:13:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:06.930+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:06.931268+0000) 2024-02-19T20:13:07.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:07.400+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:07.401246+0000) 2024-02-19T20:13:07.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:07.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:07.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:07.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:07.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:07.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:07.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:07.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:07.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:07.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:07.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:07.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:07.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:07.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:07.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:07.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:07.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:07.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:07.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:07.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:07.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:07.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:07.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:07.858+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:08.144+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:08.145800+0000) 2024-02-19T20:13:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:07.932+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:07.933854+0000) 2024-02-19T20:13:08.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:08.403+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:08.404425+0000) 2024-02-19T20:13:08.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:08.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:08.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:08.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:08.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:08.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:08.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:08.492+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:08.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:08.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:08.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:08.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:08.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:08.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:08.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:08.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:08.836+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:08.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:08.836+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:09.147+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.148283+0000) 2024-02-19T20:13:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:08.935+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:08.936174+0000) 2024-02-19T20:13:09.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:09.405+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.406498+0000) 2024-02-19T20:13:09.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:09.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:09.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:09.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:09.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:09.524+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:09.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:09.525+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:09.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:09.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:09.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:09.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:09.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:09.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:09.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:09.820+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:09.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:09.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:09.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:09.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:09.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:09.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:09.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:09.852+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:10.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:10.149+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:10.150776+0000) 2024-02-19T20:13:10.196 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:09.937+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:09.938636+0000) 2024-02-19T20:13:10.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:10.407+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:10.408453+0000) 2024-02-19T20:13:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:10.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:10.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:11.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:10.937+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:10.938823+0000) 2024-02-19T20:13:11.140 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:10.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:11.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:10.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:11.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:10.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:11.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:10.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:11.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:10.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:11.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:10.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:11.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:10.859+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:11.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:10.859+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:11.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:11.149+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.150975+0000) 2024-02-19T20:13:11.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:11.407+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.408645+0000) 2024-02-19T20:13:11.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:11.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:11.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:11.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:11.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:11.571+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:11.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:11.571+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:11.938 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:11.744+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56931 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:11.938 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:11.741+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.742385+0000) 2024-02-19T20:13:11.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:11.741+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.742439+0000) 2024-02-19T20:13:11.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:11.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:11.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:11.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:11.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:11.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:11.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:11.811+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:11.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:11.741+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.742817+0000) 2024-02-19T20:13:11.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:11.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:11.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:11.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:11.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:11.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:11.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:11.846+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:12.150+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:12.151159+0000) 2024-02-19T20:13:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:11.937+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:11.939004+0000) 2024-02-19T20:13:12.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:12.407+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:12.408843+0000) 2024-02-19T20:13:12.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:12.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:12.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:12.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:12.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:12.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:12.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:12.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:12.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:12.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:12.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:12.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:12.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:12.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:12.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:12.823+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:12.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:12.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:12.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:12.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:12.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:12.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:12.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:12.831+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:13.152+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:13.153218+0000) 2024-02-19T20:13:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:12.941+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:12.942583+0000) 2024-02-19T20:13:13.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:13.409+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:13.410917+0000) 2024-02-19T20:13:13.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:13.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:13.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:13.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:13.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:13.581+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:13.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:13.581+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:13.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:13.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:13.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:13.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:13.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:13.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:13.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:13.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:13.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:13.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:13.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:13.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:13.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:13.818+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:13.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:13.819+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:14.154+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.155954+0000) 2024-02-19T20:13:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:13.944+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:13.945063+0000) 2024-02-19T20:13:14.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:14.411+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.412980+0000) 2024-02-19T20:13:14.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:14.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:14.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:14.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:14.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:14.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:14.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:14.583+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:14.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:14.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:14.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:14.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:14.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:14.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:14.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:14.854+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:14.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:14.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:14.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:14.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:14.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:14.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:14.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:14.824+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:15.157+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:15.159010+0000) 2024-02-19T20:13:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:14.946+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:14.947663+0000) 2024-02-19T20:13:15.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:15.414+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:15.415125+0000) 2024-02-19T20:13:15.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:15.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:15.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:15.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:15.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:15.609+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:15.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:15.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:15.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:15.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:15.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:15.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:15.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:15.849+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:15.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:15.850+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:15.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:15.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:15.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:15.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:15.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:15.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:15.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:15.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:16.158+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.159240+0000) 2024-02-19T20:13:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:15.946+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:15.947852+0000) 2024-02-19T20:13:16.665 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:16.414+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.415399+0000) 2024-02-19T20:13:16.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:16.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:16.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:16.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:16.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:16.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:16.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:16.561+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:16.938 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:16.751+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 56969 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:16.939 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:16.749+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.750556+0000) 2024-02-19T20:13:16.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:16.749+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.750245+0000) 2024-02-19T20:13:16.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:16.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:16.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:16.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:16.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:16.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:16.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:16.817+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:16.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:16.748+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.749446+0000) 2024-02-19T20:13:16.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:16.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:16.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:16.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:16.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:16.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:16.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:16.892+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:17.158+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:17.159432+0000) 2024-02-19T20:13:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:16.947+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:16.948067+0000) 2024-02-19T20:13:17.665 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:17.414+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:17.415598+0000) 2024-02-19T20:13:17.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:17.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:17.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:17.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:17.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:17.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:17.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:17.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:17.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:17.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:17.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:17.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:17.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:17.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:17.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:17.869+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:17.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:17.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:17.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:17.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:17.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:17.828+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:17.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:17.829+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:18.161+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:18.162979+0000) 2024-02-19T20:13:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:17.949+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:17.950159+0000) 2024-02-19T20:13:18.665 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:18.416+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:18.417596+0000) 2024-02-19T20:13:18.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:18.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:18.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:18.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:18.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:18.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:18.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:18.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:18.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:18.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:18.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:18.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:18.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:18.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:18.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:18.850+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:18.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:18.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:18.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:18.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:18.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:18.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:18.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:18.884+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:19.164+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.165055+0000) 2024-02-19T20:13:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:18.952+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:18.953060+0000) 2024-02-19T20:13:19.665 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:19.420+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.420049+0000) 2024-02-19T20:13:19.665 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:19.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:19.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:19.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:19.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:19.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:19.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:19.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:19.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:19.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:19.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:19.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:19.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:19.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:19.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:19.904+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:19.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:19.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:19.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:19.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:19.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:19.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:19.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:19.884+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:20.196 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:20.168+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:20.168620+0000) 2024-02-19T20:13:20.197 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:19.955+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:19.955104+0000) 2024-02-19T20:13:20.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:20.421+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:20.422055+0000) 2024-02-19T20:13:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:20.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:20.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:20.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:20.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:20.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:21.141 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:20.954+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:20.955266+0000) 2024-02-19T20:13:21.142 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:20.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:21.142 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:20.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:21.142 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:20.933+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:21.142 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:20.934+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:21.142 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:20.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:21.142 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:20.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:21.142 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:20.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:21.143 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:20.884+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:21.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:21.167+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.168958+0000) 2024-02-19T20:13:21.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:21.421+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.422224+0000) 2024-02-19T20:13:21.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:21.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:21.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:21.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:21.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:21.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:21.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:21.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:21.996 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:21.758+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57010 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:21.996 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:21.755+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.756645+0000) 2024-02-19T20:13:21.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:21.954+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.955480+0000) 2024-02-19T20:13:21.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:21.756+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.757407+0000) 2024-02-19T20:13:21.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:21.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:21.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:21.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:21.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:21.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:21.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:21.952+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:21.998 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:21.756+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:21.757579+0000) 2024-02-19T20:13:21.998 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:21.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:21.998 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:21.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:21.998 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:21.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:21.998 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:21.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:22.168+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:22.169223+0000) 2024-02-19T20:13:22.665 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:22.421+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:22.422419+0000) 2024-02-19T20:13:22.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:22.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:22.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:22.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:22.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:22.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:22.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:22.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:22.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:22.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:22.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:22.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:22.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:22.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:22.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:22.906+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:22.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:22.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:22.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:22.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:22.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:22.823+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:22.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:22.824+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:23.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:23.170+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:23.171873+0000) 2024-02-19T20:13:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:22.956+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:22.957477+0000) 2024-02-19T20:13:23.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:23.424+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:23.425186+0000) 2024-02-19T20:13:23.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:23.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:23.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:23.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:23.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:23.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:23.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:23.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:23.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:23.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:23.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:23.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:23.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:23.843+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:23.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:23.844+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:24.173+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.174350+0000) 2024-02-19T20:13:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:23.959+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:23.960066+0000) 2024-02-19T20:13:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:23.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:23.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:23.947+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:24.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:23.948+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:24.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:24.427+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.428789+0000) 2024-02-19T20:13:24.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:24.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:24.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:24.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:24.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:24.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:24.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:24.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:24.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:24.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:24.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:24.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:24.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:24.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:24.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:24.872+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:25.175+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:25.176329+0000) 2024-02-19T20:13:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:24.962+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:24.963456+0000) 2024-02-19T20:13:25.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:24.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:24.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:24.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:24.967+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:25.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:25.429+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:25.430760+0000) 2024-02-19T20:13:25.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:25.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:25.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:25.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:25.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:25.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:25.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:25.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:25.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:25.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:25.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:25.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:25.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:25.861+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:25.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:25.862+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:26.175+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.176535+0000) 2024-02-19T20:13:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:25.962+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:25.963645+0000) 2024-02-19T20:13:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:25.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:25.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:26.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:25.964+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:26.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:25.965+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:26.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:26.429+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.430968+0000) 2024-02-19T20:13:26.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:26.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:26.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:26.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:26.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:26.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:26.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:26.480+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:26.939 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:26.765+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57048 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:26.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:26.763+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.764192+0000) 2024-02-19T20:13:26.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:26.762+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.763482+0000) 2024-02-19T20:13:26.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:26.763+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.764409+0000) 2024-02-19T20:13:26.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:26.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:26.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:26.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:26.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:26.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:26.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:26.823+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:27.175+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:27.176729+0000) 2024-02-19T20:13:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:26.962+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:26.963835+0000) 2024-02-19T20:13:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:27.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:27.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:27.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:27.002+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:27.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:27.430+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:27.431164+0000) 2024-02-19T20:13:27.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:27.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:27.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:27.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:27.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:27.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:27.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:27.469+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:27.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:27.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:27.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:27.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:27.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:27.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:27.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:27.820+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:28.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:27.964+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:27.965890+0000) 2024-02-19T20:13:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:28.177+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:28.178851+0000) 2024-02-19T20:13:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:28.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:28.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:28.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:28.037+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:28.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:28.432+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:28.433194+0000) 2024-02-19T20:13:28.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:28.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:28.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:28.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:28.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:28.461+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:28.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:28.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:28.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:28.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:28.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:28.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:28.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:28.805+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:28.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:28.806+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:29.179+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.180942+0000) 2024-02-19T20:13:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:28.968+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:28.969506+0000) 2024-02-19T20:13:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:29.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:29.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:29.065+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:29.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:29.066+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:29.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:29.435+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.436726+0000) 2024-02-19T20:13:29.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:29.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:29.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:29.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:29.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:29.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:29.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:29.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:29.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:29.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:29.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:29.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:29.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:29.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:29.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:29.826+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:30.197 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:29.970+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:29.971531+0000) 2024-02-19T20:13:30.197 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:30.182+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:30.183114+0000) 2024-02-19T20:13:30.197 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:30.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:30.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:30.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:30.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:30.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:30.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:30.082+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:30.437+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:30.438796+0000) 2024-02-19T20:13:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:30.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:30.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:30.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:30.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:30.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:30.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:30.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:30.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:30.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:30.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:30.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:30.849+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:31.182+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.183276+0000) 2024-02-19T20:13:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:30.970+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:30.971737+0000) 2024-02-19T20:13:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:31.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:31.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:31.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:31.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:31.086+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:31.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:31.437+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.438987+0000) 2024-02-19T20:13:31.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:31.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:31.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:31.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:31.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:31.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:31.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:31.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:31.940 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:31.772+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57089 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:31.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:31.769+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.770668+0000) 2024-02-19T20:13:31.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:31.768+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.769812+0000) 2024-02-19T20:13:31.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:31.769+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.770305+0000) 2024-02-19T20:13:31.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:31.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:31.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:31.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:31.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:31.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:31.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:31.865+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:32.182+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:32.183438+0000) 2024-02-19T20:13:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:31.970+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:31.971967+0000) 2024-02-19T20:13:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:32.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:32.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:32.111+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:32.111+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:32.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:32.438+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:32.439202+0000) 2024-02-19T20:13:32.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:32.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:32.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:32.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:32.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:32.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:32.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:32.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:32.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:32.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:32.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:32.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:32.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:32.873+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:33.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:33.185+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:33.186946+0000) 2024-02-19T20:13:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:32.972+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:32.974005+0000) 2024-02-19T20:13:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:33.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:33.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:33.139+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:33.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:33.140+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:33.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:33.440+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:33.441704+0000) 2024-02-19T20:13:33.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:33.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:33.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:33.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:33.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:33.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:33.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:33.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:33.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:33.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:33.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:33.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:33.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:33.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:33.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:33.834+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:34.188+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.189148+0000) 2024-02-19T20:13:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:33.975+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:33.976079+0000) 2024-02-19T20:13:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:34.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:34.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:34.130+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:34.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:34.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:34.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:34.444+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.445244+0000) 2024-02-19T20:13:34.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:34.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:34.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:34.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:34.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:34.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:34.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:34.380+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:34.940 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:34.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:34.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:34.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:34.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:34.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:34.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:34.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:35.192 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:34.977+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:34.978085+0000) 2024-02-19T20:13:35.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:35.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:35.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:35.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:35.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:35.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:35.193 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:35.108+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:35.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:35.191+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:35.192714+0000) 2024-02-19T20:13:35.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:35.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:35.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:35.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:35.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:35.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:35.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:35.351+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:35.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:35.446+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:35.447354+0000) 2024-02-19T20:13:35.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:35.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:35.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:35.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:35.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:35.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:35.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:35.885+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:36.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:36.191+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.192969+0000) 2024-02-19T20:13:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:35.977+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:35.978278+0000) 2024-02-19T20:13:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:36.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:36.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:36.151+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:36.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:36.152+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:36.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:36.446+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.447546+0000) 2024-02-19T20:13:36.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:36.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:36.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:36.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:36.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:36.378+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:36.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:36.379+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:36.941 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:36.778+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57127 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:36.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:36.775+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.776643+0000) 2024-02-19T20:13:36.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:36.775+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.776297+0000) 2024-02-19T20:13:36.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:36.775+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.776908+0000) 2024-02-19T20:13:36.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:36.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:36.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:36.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:36.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:36.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:36.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:36.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:37.193 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:36.977+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:36.978491+0000) 2024-02-19T20:13:37.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:37.192+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:37.193191+0000) 2024-02-19T20:13:37.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:37.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:37.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:37.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:37.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:37.192+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:37.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:37.192+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:37.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:37.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:37.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:37.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:37.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:37.380+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:37.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:37.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:37.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:37.446+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:37.447768+0000) 2024-02-19T20:13:37.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:37.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:37.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:37.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:37.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:37.863+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:37.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:37.864+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:38.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:37.980+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:37.981927+0000) 2024-02-19T20:13:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:38.194+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:38.195620+0000) 2024-02-19T20:13:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:38.191+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:38.191+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:38.191+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:38.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:38.192+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:38.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:38.448+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:38.449903+0000) 2024-02-19T20:13:38.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:38.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:38.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:38.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:38.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:38.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:38.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:38.400+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:38.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:38.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:38.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:38.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:38.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:38.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:38.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:38.908+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:39.198 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:38.984+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:38.985820+0000) 2024-02-19T20:13:39.452 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:39.196+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.197905+0000) 2024-02-19T20:13:39.452 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:39.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:39.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:39.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:39.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:39.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:39.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:39.402+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:39.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:39.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:39.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:39.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:39.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:39.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:39.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:39.208+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:39.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:39.450+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.451934+0000) 2024-02-19T20:13:39.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:39.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:39.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:39.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:39.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:39.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:39.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:39.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:40.199+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:40.200148+0000) 2024-02-19T20:13:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:39.986+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:39.987989+0000) 2024-02-19T20:13:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:40.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:40.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:40.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:40.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:40.183+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:40.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:40.454+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:40.455572+0000) 2024-02-19T20:13:40.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:40.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:40.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:40.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:40.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:40.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:40.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:40.370+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:40.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:40.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:40.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:40.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:40.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:40.866+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:40.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:40.867+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:41.200 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:40.987+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:40.988199+0000) 2024-02-19T20:13:41.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:41.199+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.200308+0000) 2024-02-19T20:13:41.456 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:41.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:41.456 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:41.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:41.456 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:41.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:41.457 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:41.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:41.457 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:41.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:41.457 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:41.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:41.457 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:41.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:41.457 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:41.205+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:41.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:41.454+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.455850+0000) 2024-02-19T20:13:41.988 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:41.786+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57168 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:41.989 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:41.782+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.783946+0000) 2024-02-19T20:13:41.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:41.783+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.784462+0000) 2024-02-19T20:13:41.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:41.784+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.785081+0000) 2024-02-19T20:13:41.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:41.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:41.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:41.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:41.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:41.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:41.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:41.819+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:42.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:42.199+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:42.200514+0000) 2024-02-19T20:13:42.243 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:41.987+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:41.988378+0000) 2024-02-19T20:13:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:42.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:42.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:42.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:42.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:42.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:42.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:42.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:42.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:42.455+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:42.456043+0000) 2024-02-19T20:13:42.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:42.850+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:42.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:42.850+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:42.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:42.850+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:42.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:42.851+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:43.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:42.989+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:42.990629+0000) 2024-02-19T20:13:43.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:43.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:43.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:43.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:43.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:43.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:43.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:43.253+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:43.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:43.201+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:43.202600+0000) 2024-02-19T20:13:43.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:43.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:43.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:43.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:43.460 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:43.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:43.460 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:43.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:43.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:43.457+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:43.458139+0000) 2024-02-19T20:13:43.994 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:43.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:43.994 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:43.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:43.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:43.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:43.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:43.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:44.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:44.203+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.204956+0000) 2024-02-19T20:13:44.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:43.993+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:43.994224+0000) 2024-02-19T20:13:44.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:44.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:44.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:44.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:44.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:44.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:44.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:44.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:44.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:44.460+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.461672+0000) 2024-02-19T20:13:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:44.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:44.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:44.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:44.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:44.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:44.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:44.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:44.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:44.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:44.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:44.942 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:44.897+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:45.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:44.995+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:44.996401+0000) 2024-02-19T20:13:45.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:45.206+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:45.207023+0000) 2024-02-19T20:13:45.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:45.203+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:45.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:45.203+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:45.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:45.203+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:45.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:45.204+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:45.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:45.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:45.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:45.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:45.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:45.277+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:45.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:45.278+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:45.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:45.462+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:45.463894+0000) 2024-02-19T20:13:45.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:45.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:45.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:45.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:45.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:45.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:45.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:45.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:46.206+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.207202+0000) 2024-02-19T20:13:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:45.995+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:45.996562+0000) 2024-02-19T20:13:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:46.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:46.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:46.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:46.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:46.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:46.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:46.463+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.464082+0000) 2024-02-19T20:13:46.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:46.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:46.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:46.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:46.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:46.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:46.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:46.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:46.942 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:46.793+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57206 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:46.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:46.789+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.790967+0000) 2024-02-19T20:13:46.943 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:46.790+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.791760+0000) 2024-02-19T20:13:46.943 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:46.790+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.791330+0000) 2024-02-19T20:13:47.203 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:46.995+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:46.996745+0000) 2024-02-19T20:13:47.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:46.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:47.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:46.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:47.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:46.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:47.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:46.954+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:47.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:47.206+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:47.207367+0000) 2024-02-19T20:13:47.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:47.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:47.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:47.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:47.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:47.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:47.465 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:47.276+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:47.465 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:47.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:47.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:47.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:47.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:47.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:47.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:47.202+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:47.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:47.463+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:47.464284+0000) 2024-02-19T20:13:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:47.997+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:47.998802+0000) 2024-02-19T20:13:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:48.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:48.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:48.224+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:48.225+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:47.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:47.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:47.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:47.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:48.208+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:48.209515+0000) 2024-02-19T20:13:48.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:48.465+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:48.466801+0000) 2024-02-19T20:13:48.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:48.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:48.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:48.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:48.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:48.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:48.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:48.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:49.194 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:49.001+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.002427+0000) 2024-02-19T20:13:49.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:48.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:49.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:48.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:49.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:48.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:49.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:48.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:49.469 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:49.210+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.211717+0000) 2024-02-19T20:13:49.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:49.193+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:49.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:49.193+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:49.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:49.193+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:49.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:49.194+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:49.470 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:49.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:49.470 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:49.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:49.470 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:49.366+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:49.470 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:49.366+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:49.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:49.467+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:49.468916+0000) 2024-02-19T20:13:50.243 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:50.004+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:50.005087+0000) 2024-02-19T20:13:50.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:49.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:50.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:49.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:50.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:49.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:50.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:49.981+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:50.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:50.212+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:50.213775+0000) 2024-02-19T20:13:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:50.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:50.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:50.241+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:50.242+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:50.470+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:50.471262+0000) 2024-02-19T20:13:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:50.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:50.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:50.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:50.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:50.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:51.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:51.004+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.005268+0000) 2024-02-19T20:13:51.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:50.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:51.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:50.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:51.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:50.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:51.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:50.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:51.471 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:51.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:51.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:51.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:51.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:51.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:51.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:51.316+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:51.472 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:51.212+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.213949+0000) 2024-02-19T20:13:51.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:51.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:51.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:51.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:51.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:51.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:51.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:51.262+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:51.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:51.470+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.471550+0000) 2024-02-19T20:13:51.977 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:51.800+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57247 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:51.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:51.797+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.798774+0000) 2024-02-19T20:13:51.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:51.798+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.799260+0000) 2024-02-19T20:13:51.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:51.796+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:51.797995+0000) 2024-02-19T20:13:51.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:51.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:51.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:51.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:51.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:51.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:52.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:51.977+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:52.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:52.213+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:52.214173+0000) 2024-02-19T20:13:52.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:52.004+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:52.005435+0000) 2024-02-19T20:13:52.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:52.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:52.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:52.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:52.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:52.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:52.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:52.268+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:52.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:52.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:52.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:52.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:52.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:52.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:52.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:52.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:52.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:52.470+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:52.471718+0000) 2024-02-19T20:13:53.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:53.007+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:53.008119+0000) 2024-02-19T20:13:53.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:52.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:53.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:52.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:53.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:52.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:53.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:52.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:53.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:53.216+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:53.217729+0000) 2024-02-19T20:13:53.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:53.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:53.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:53.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:53.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:53.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:53.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:53.272+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:53.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:53.472+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:53.473711+0000) 2024-02-19T20:13:53.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:53.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:53.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:53.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:53.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:53.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:53.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:53.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:53.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:53.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:53.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:53.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:53.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:53.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:53.977 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:53.975+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:54.233 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:54.009+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.010085+0000) 2024-02-19T20:13:54.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:54.219+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.220193+0000) 2024-02-19T20:13:54.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:54.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:54.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:54.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:54.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:54.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:54.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:54.232+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:54.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:54.474+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:54.476000+0000) 2024-02-19T20:13:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:54.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:54.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:54.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:54.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:54.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:55.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:55.011+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:55.012066+0000) 2024-02-19T20:13:55.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:54.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:55.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:54.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:55.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:54.951+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:55.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:54.952+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:55.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:55.221+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:55.222228+0000) 2024-02-19T20:13:55.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:55.250+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:55.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:55.250+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:55.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:55.250+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:55.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:55.251+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:55.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:55.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:55.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:55.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:55.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:55.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:55.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:55.278+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:55.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:55.477+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:55.478083+0000) 2024-02-19T20:13:56.209 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:56.011+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.012263+0000) 2024-02-19T20:13:56.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:55.976+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:56.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:55.976+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:56.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:55.976+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:56.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:55.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:56.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:56.221+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.222395+0000) 2024-02-19T20:13:56.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:56.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:56.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:56.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:56.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:56.208+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:56.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:56.208+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:56.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:56.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:56.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:56.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:56.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:56.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:56.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:56.309+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:56.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:56.477+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.478266+0000) 2024-02-19T20:13:57.145 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:56.804+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.805924+0000) 2024-02-19T20:13:57.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:13:56.806+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57285 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:13:57.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:56.804+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.805207+0000) 2024-02-19T20:13:57.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:57.011+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:57.012455+0000) 2024-02-19T20:13:57.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:56.804+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:56.805676+0000) 2024-02-19T20:13:57.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:56.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:57.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:56.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:57.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:56.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:57.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:56.956+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:57.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:57.221+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:57.222583+0000) 2024-02-19T20:13:57.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:57.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:57.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:57.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:57.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:57.231+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:57.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:57.232+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:57.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:57.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:57.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:57.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:57.480 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:57.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:57.480 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:57.342+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:57.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:57.477+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:57.478450+0000) 2024-02-19T20:13:58.191 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:58.013+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:58.014630+0000) 2024-02-19T20:13:58.192 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:57.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:58.192 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:57.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:58.192 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:57.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:58.192 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:57.939+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:58.482 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:58.225+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:58.226192+0000) 2024-02-19T20:13:58.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:58.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:58.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:58.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:58.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:58.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:58.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:58.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:58.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:58.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:58.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:58.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:58.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:58.190+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:58.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:58.191+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:58.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:58.480+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:58.481777+0000) 2024-02-19T20:13:59.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:59.015+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.016812+0000) 2024-02-19T20:13:59.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:58.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:59.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:58.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:59.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:58.900+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:13:59.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:58.901+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:13:59.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:59.227+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.228184+0000) 2024-02-19T20:13:59.486 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:59.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:59.486 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:59.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:59.486 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:59.237+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:13:59.486 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:59.238+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:59.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:59.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:59.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:59.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:13:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:13:59.307+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:13:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:13:59.484+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:13:59.485288+0000) 2024-02-19T20:14:00.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:00.017+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:00.018836+0000) 2024-02-19T20:14:00.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:59.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:00.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:59.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:00.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:59.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:00.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:13:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:13:59.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:00.396 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:00.229+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:00.230812+0000) 2024-02-19T20:14:00.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:00.268+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:00.396 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:00.268+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:00.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:00.268+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:00.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:00.269+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:00.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:00.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:00.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:00.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:00.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:00.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:00.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:00.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:00.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:00.487+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:00.488653+0000) 2024-02-19T20:14:00.943 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:00.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:00.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:00.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:00.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:00.868+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:00.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:00.869+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:01.231 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:01.018+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.019101+0000) 2024-02-19T20:14:01.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:01.230+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.231051+0000) 2024-02-19T20:14:01.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:01.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:01.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:01.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:01.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:01.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:01.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:01.279+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:01.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:01.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:01.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:01.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:01.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:01.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:01.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:01.282+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:01.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:01.487+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.488847+0000) 2024-02-19T20:14:02.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:01.814+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57326 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:02.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:01.811+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.812644+0000) 2024-02-19T20:14:02.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:01.811+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.812390+0000) 2024-02-19T20:14:02.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:02.018+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:02.019266+0000) 2024-02-19T20:14:02.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:01.812+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:01.813297+0000) 2024-02-19T20:14:02.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:01.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:02.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:01.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:02.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:01.903+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:02.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:01.904+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:02.396 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:02.230+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:02.231228+0000) 2024-02-19T20:14:02.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:02.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:02.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:02.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:02.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:02.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:02.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:02.330+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:02.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:02.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:02.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:02.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:02.398 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:02.240+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:02.398 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:02.241+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:02.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:02.488+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:02.489063+0000) 2024-02-19T20:14:03.233 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:03.020+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:03.021456+0000) 2024-02-19T20:14:03.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:02.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:03.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:02.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:03.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:02.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:03.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:02.939+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:03.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:03.232+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:03.233240+0000) 2024-02-19T20:14:03.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:03.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:03.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:03.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:03.258+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:03.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:03.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:03.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:03.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:03.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:03.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:03.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:03.307+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:03.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:03.490+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:03.491090+0000) 2024-02-19T20:14:04.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:04.022+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.023444+0000) 2024-02-19T20:14:04.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:03.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:04.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:03.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:04.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:03.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:04.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:03.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:04.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:04.234+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.235352+0000) 2024-02-19T20:14:04.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:04.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:04.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:04.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:04.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:04.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:04.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:04.285+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:04.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:04.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:04.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:04.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:04.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:04.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:04.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:04.341+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:04.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:04.493+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:04.494775+0000) 2024-02-19T20:14:05.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:05.024+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:05.025404+0000) 2024-02-19T20:14:05.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:04.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:05.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:04.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:05.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:04.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:05.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:04.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:05.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:05.237+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:05.237513+0000) 2024-02-19T20:14:05.399 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:05.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:05.399 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:05.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:05.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:05.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:05.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:05.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:05.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:05.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:05.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:05.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:05.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:05.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:05.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:05.254+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:05.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:05.497+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:05.497101+0000) 2024-02-19T20:14:05.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:05.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:05.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:05.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:05.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:05.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:05.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:05.882+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:06.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:06.025+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.025636+0000) 2024-02-19T20:14:06.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:06.237+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.237840+0000) 2024-02-19T20:14:06.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:06.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:06.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:06.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:06.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:06.311+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:06.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:06.312+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:06.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:06.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:06.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:06.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:06.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:06.276+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:06.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:06.277+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:06.496+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.497296+0000) 2024-02-19T20:14:07.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:06.821+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57364 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:07.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:06.819+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.820976+0000) 2024-02-19T20:14:07.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:06.819+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.820264+0000) 2024-02-19T20:14:07.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:07.024+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:07.025800+0000) 2024-02-19T20:14:07.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:06.818+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:06.819864+0000) 2024-02-19T20:14:07.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:06.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:07.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:06.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:07.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:06.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:07.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:06.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:07.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:07.237+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:07.238125+0000) 2024-02-19T20:14:07.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:07.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:07.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:07.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:07.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:07.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:07.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:07.356+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:07.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:07.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:07.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:07.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:07.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:07.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:07.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:07.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:07.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:07.496+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:07.497494+0000) 2024-02-19T20:14:07.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:07.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:07.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:07.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:07.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:07.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:07.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:07.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:08.241 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:08.026+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:08.027789+0000) 2024-02-19T20:14:08.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:08.240+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:08.241644+0000) 2024-02-19T20:14:08.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:08.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:08.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:08.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:08.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:08.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:08.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:08.264+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:08.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:08.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:08.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:08.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:08.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:08.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:08.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:08.343+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:08.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:08.499+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:08.500803+0000) 2024-02-19T20:14:09.244 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:09.028+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.029836+0000) 2024-02-19T20:14:09.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:08.948+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:09.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:08.948+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:09.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:08.948+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:09.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:08.949+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:09.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:09.242+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.243879+0000) 2024-02-19T20:14:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:09.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:09.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:09.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:09.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:09.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:09.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:09.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:09.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:09.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:09.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:09.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:09.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:09.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:09.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:09.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:09.503+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:09.504493+0000) 2024-02-19T20:14:10.199 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:10.030+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:10.031928+0000) 2024-02-19T20:14:10.199 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:09.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:10.199 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:09.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:10.200 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:09.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:10.200 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:09.927+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:10.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:10.246+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:10.247327+0000) 2024-02-19T20:14:10.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:10.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:10.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:10.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:10.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:10.326+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:10.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:10.327+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:10.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:10.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:10.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:10.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:10.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:10.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:10.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:10.336+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:10.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:10.506+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:10.507965+0000) 2024-02-19T20:14:11.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:11.031+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.032151+0000) 2024-02-19T20:14:11.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:10.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:11.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:10.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:11.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:10.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:11.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:10.965+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:11.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:11.246+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.247543+0000) 2024-02-19T20:14:11.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:11.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:11.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:11.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:11.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:11.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:11.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:11.366+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:11.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:11.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:11.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:11.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:11.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:11.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:11.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:11.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:11.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:11.507+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.508234+0000) 2024-02-19T20:14:12.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:11.828+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57405 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:12.147 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:11.825+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.826492+0000) 2024-02-19T20:14:12.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:11.826+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.827163+0000) 2024-02-19T20:14:12.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:12.031+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:12.032336+0000) 2024-02-19T20:14:12.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:11.825+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:11.826625+0000) 2024-02-19T20:14:12.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:11.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:12.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:11.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:12.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:11.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:12.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:11.983+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:12.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:12.246+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:12.247739+0000) 2024-02-19T20:14:12.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:12.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:12.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:12.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:12.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:12.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:12.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:12.343+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:12.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:12.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:12.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:12.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:12.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:12.315+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:12.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:12.316+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:12.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:12.507+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:12.508453+0000) 2024-02-19T20:14:13.250 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:13.034+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:13.035782+0000) 2024-02-19T20:14:13.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:13.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:13.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:13.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:13.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:13.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:13.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:13.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:13.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:13.248+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:13.249889+0000) 2024-02-19T20:14:13.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:13.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:13.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:13.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:13.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:13.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:13.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:13.371+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:13.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:13.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:13.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:13.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:13.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:13.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:13.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:13.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:13.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:13.511+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:13.512082+0000) 2024-02-19T20:14:14.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:14.036+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.037849+0000) 2024-02-19T20:14:14.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:14.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:14.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:14.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:14.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:14.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:14.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:14.020+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:14.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:14.251+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.252357+0000) 2024-02-19T20:14:14.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:14.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:14.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:14.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:14.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:14.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:14.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:14.415+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:14.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:14.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:14.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:14.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:14.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:14.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:14.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:14.362+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:14.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:14.513+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:14.514565+0000) 2024-02-19T20:14:15.255 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:15.038+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:15.039931+0000) 2024-02-19T20:14:15.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:15.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:15.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:15.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:15.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.253+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:15.254872+0000) 2024-02-19T20:14:15.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:15.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:15.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:15.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:15.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:15.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:15.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:15.402+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:15.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:15.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:15.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:15.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:15.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:15.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:15.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:15.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:15.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:15.515+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:15.516871+0000) 2024-02-19T20:14:16.255 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:16.039+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.040185+0000) 2024-02-19T20:14:16.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:16.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:16.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:16.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:15.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:16.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:16.254+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.255120+0000) 2024-02-19T20:14:16.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:16.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:16.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:16.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:16.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:16.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:16.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:16.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:16.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:16.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:16.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:16.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:16.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:16.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:16.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:16.432+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:16.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:16.516+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.517129+0000) 2024-02-19T20:14:17.147 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:16.835+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57443 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:17.147 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:16.831+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.832746+0000) 2024-02-19T20:14:17.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:16.832+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.833697+0000) 2024-02-19T20:14:17.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:17.039+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:17.040366+0000) 2024-02-19T20:14:17.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:16.832+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:16.833685+0000) 2024-02-19T20:14:17.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:16.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:17.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:16.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:17.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:16.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:17.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:16.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:17.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:17.254+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:17.255307+0000) 2024-02-19T20:14:17.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:17.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:17.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:17.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:17.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:17.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:17.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:17.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:17.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:17.516+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:17.517291+0000) 2024-02-19T20:14:17.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:17.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:17.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:17.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:17.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:17.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:17.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:17.424+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:18.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:18.042+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:18.043895+0000) 2024-02-19T20:14:18.259 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:17.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:18.259 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:17.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:18.260 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:17.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:18.260 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:17.994+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:18.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:18.258+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:18.259035+0000) 2024-02-19T20:14:18.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:18.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:18.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:18.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:18.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:18.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:18.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:18.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:18.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:18.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:18.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:18.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:18.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:18.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:18.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:18.425+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:18.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:18.519+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:18.520882+0000) 2024-02-19T20:14:19.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:19.045+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.046242+0000) 2024-02-19T20:14:19.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:18.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:19.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:18.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:19.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:18.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:19.262 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:18.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:19.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:19.260+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.261268+0000) 2024-02-19T20:14:19.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:19.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:19.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:19.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:19.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:19.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:19.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:19.432+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:19.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:19.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:19.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:19.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:19.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:19.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:19.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:19.403+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:19.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:19.522+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:19.523069+0000) 2024-02-19T20:14:20.200 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:20.047+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:20.048402+0000) 2024-02-19T20:14:20.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:19.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:20.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:19.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:20.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:19.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:20.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:19.946+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:20.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:20.264+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:20.265183+0000) 2024-02-19T20:14:20.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:20.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:20.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:20.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:20.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:20.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:20.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:20.391+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:20.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:20.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:20.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:20.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:20.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:20.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:20.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:20.402+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:20.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:20.524+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:20.525625+0000) 2024-02-19T20:14:21.265 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:21.047+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.048649+0000) 2024-02-19T20:14:21.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:20.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:21.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:20.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:21.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:20.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:21.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:20.954+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:21.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:21.264+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.265540+0000) 2024-02-19T20:14:21.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:21.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:21.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:21.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:21.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:21.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:21.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:21.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:21.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:21.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:21.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:21.393+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:21.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:21.524+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.525966+0000) 2024-02-19T20:14:22.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:21.842+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57484 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:22.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:21.839+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.840689+0000) 2024-02-19T20:14:22.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:21.839+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.840905+0000) 2024-02-19T20:14:22.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:22.047+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:22.048850+0000) 2024-02-19T20:14:22.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:21.840+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:21.841405+0000) 2024-02-19T20:14:22.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:21.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:22.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:21.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:22.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:21.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:22.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:21.969+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:22.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:22.264+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:22.265730+0000) 2024-02-19T20:14:22.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:22.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:22.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:22.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:22.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:22.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:22.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:22.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:22.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:22.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:22.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:22.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:22.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:22.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:22.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:22.400+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:22.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:22.525+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:22.526185+0000) 2024-02-19T20:14:23.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:23.050+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:23.051087+0000) 2024-02-19T20:14:23.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:22.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:23.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:22.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:23.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:22.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:23.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:22.969+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:23.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:23.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:23.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:23.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:23.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:23.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:23.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:23.357+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:23.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:23.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:23.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:23.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:23.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:23.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:23.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:23.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:23.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:23.266+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:23.267823+0000) 2024-02-19T20:14:23.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:23.527+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:23.528166+0000) 2024-02-19T20:14:24.270 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:24.052+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.053168+0000) 2024-02-19T20:14:24.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:24.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:24.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:24.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:24.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.269+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.270304+0000) 2024-02-19T20:14:24.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:24.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:24.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:24.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:24.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:24.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:24.532 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:24.314+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:24.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:24.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:24.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:24.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:24.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:24.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:24.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:24.357+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:24.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:24.529+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:24.530763+0000) 2024-02-19T20:14:25.273 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:25.055+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:25.056871+0000) 2024-02-19T20:14:25.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:25.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:25.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:25.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:24.982+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:25.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:25.271+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:25.272959+0000) 2024-02-19T20:14:25.533 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:25.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:25.534 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:25.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:25.534 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:25.353+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:25.534 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:25.354+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:25.534 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:25.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:25.534 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:25.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:25.534 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:25.393+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:25.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:25.393+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:25.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:25.532+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:25.533271+0000) 2024-02-19T20:14:26.273 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:26.056+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.057177+0000) 2024-02-19T20:14:26.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:25.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:26.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:25.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:26.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:25.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:26.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:25.959+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:26.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:26.272+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.273226+0000) 2024-02-19T20:14:26.534 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:26.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:26.534 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:26.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:26.534 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:26.380+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:26.534 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:26.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:26.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:26.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:26.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:26.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:26.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:26.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:26.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:26.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:26.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:26.532+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.533573+0000) 2024-02-19T20:14:27.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:26.847+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.848228+0000) 2024-02-19T20:14:27.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:26.849+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57522 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:27.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:26.848+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.849112+0000) 2024-02-19T20:14:27.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:27.056+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:27.057342+0000) 2024-02-19T20:14:27.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:26.846+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:26.847473+0000) 2024-02-19T20:14:27.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:26.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:27.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:26.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:27.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:26.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:27.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:26.987+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:27.415 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:27.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:27.415 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:27.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:27.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:27.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:27.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:27.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:27.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:27.272+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:27.273426+0000) 2024-02-19T20:14:27.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:27.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:27.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:27.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:27.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:27.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:27.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:27.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:27.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:27.532+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:27.533763+0000) 2024-02-19T20:14:28.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:28.058+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:28.059465+0000) 2024-02-19T20:14:28.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:27.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:28.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:27.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:28.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:27.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:28.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:27.986+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:28.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:28.274+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:28.275555+0000) 2024-02-19T20:14:28.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:28.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:28.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:28.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:28.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:28.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:28.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:28.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:28.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:28.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:28.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:28.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:28.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:28.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:28.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:28.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:28.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:28.535+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:28.536234+0000) 2024-02-19T20:14:29.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:29.061+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.062910+0000) 2024-02-19T20:14:29.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:28.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:29.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:28.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:29.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:28.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:29.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:28.962+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:29.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:29.277+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.278224+0000) 2024-02-19T20:14:29.539 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:29.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:29.539 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:29.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:29.539 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:29.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:29.539 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:29.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:29.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:29.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:29.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:29.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:29.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:29.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:29.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:29.318+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:29.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:29.537+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:29.538764+0000) 2024-02-19T20:14:30.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:30.065+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:30.066452+0000) 2024-02-19T20:14:30.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:29.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:30.201 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:29.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:30.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:29.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:30.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:29.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:30.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:30.279+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:30.280730+0000) 2024-02-19T20:14:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:30.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:30.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:30.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:30.405+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:30.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:30.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:30.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:30.330+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:30.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:30.540+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:30.541114+0000) 2024-02-19T20:14:31.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:31.065+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.066783+0000) 2024-02-19T20:14:31.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:30.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:31.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:30.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:31.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:30.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:31.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:30.991+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:31.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:31.279+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.280991+0000) 2024-02-19T20:14:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:31.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:31.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:31.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:31.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:31.371+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:31.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:31.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:31.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:31.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:31.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:31.312+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:31.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:31.540+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.541377+0000) 2024-02-19T20:14:32.148 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:31.856+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57563 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:32.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:31.854+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.855201+0000) 2024-02-19T20:14:32.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:31.853+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.854642+0000) 2024-02-19T20:14:32.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:32.065+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:32.066975+0000) 2024-02-19T20:14:32.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:31.853+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:31.854912+0000) 2024-02-19T20:14:32.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:31.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:32.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:31.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:32.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:31.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:32.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:32.000+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:32.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:32.280+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:32.281181+0000) 2024-02-19T20:14:32.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:32.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:32.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:32.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:32.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:32.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:32.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:32.368+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:32.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:32.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:32.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:32.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:32.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:32.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:32.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:32.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:32.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:32.540+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:32.541547+0000) 2024-02-19T20:14:33.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:33.069+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:33.070473+0000) 2024-02-19T20:14:33.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:32.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:33.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:32.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:33.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:32.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:33.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:32.975+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:33.282+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:33.283168+0000) 2024-02-19T20:14:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:33.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:33.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:33.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:33.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:33.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:33.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:33.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:33.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:33.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:33.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:33.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:33.542+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:33.543653+0000) 2024-02-19T20:14:34.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:34.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:34.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:34.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:34.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:34.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:34.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:34.025+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:34.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:34.072+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.073929+0000) 2024-02-19T20:14:34.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:34.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:34.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:34.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:34.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:34.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:34.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:34.273+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:34.284+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.285349+0000) 2024-02-19T20:14:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:34.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:34.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:34.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:34.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:34.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:34.284+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:34.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:34.544+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:34.545831+0000) 2024-02-19T20:14:35.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:35.076+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:35.077416+0000) 2024-02-19T20:14:35.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:35.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:35.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:35.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.022+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:35.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.287+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:35.288049+0000) 2024-02-19T20:14:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:35.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:35.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:35.270+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:35.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:35.271+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:35.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:35.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:35.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:35.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:35.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:35.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:35.546+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:35.548000+0000) 2024-02-19T20:14:36.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:36.076+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.077795+0000) 2024-02-19T20:14:36.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:36.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:36.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:36.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:36.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:36.286+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:36.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:36.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:36.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:36.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:35.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:36.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:36.287+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.288281+0000) 2024-02-19T20:14:36.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:36.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:36.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:36.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:36.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:36.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:36.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:36.296+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:36.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:36.547+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.548329+0000) 2024-02-19T20:14:37.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:36.863+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57601 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:37.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:36.861+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.862148+0000) 2024-02-19T20:14:37.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:36.860+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.861825+0000) 2024-02-19T20:14:37.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:37.076+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:37.077945+0000) 2024-02-19T20:14:37.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:36.861+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:36.862388+0000) 2024-02-19T20:14:37.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:37.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:37.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:37.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:37.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:37.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:37.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:37.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:37.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:37.287+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:37.288462+0000) 2024-02-19T20:14:37.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:37.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:37.424 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:37.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:37.424 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:37.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:37.424 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:37.255+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:37.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:37.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:37.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:37.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:37.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:37.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:37.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:37.266+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:37.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:37.547+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:37.548531+0000) 2024-02-19T20:14:38.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:38.078+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:38.079988+0000) 2024-02-19T20:14:38.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:38.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:38.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:38.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:38.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:38.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:38.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:38.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:38.289+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:38.290460+0000) 2024-02-19T20:14:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:38.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:38.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:38.207+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:38.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:38.208+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:38.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:38.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:38.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:38.544 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:38.255+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:38.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:38.551+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:38.552097+0000) 2024-02-19T20:14:39.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:39.081+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.082347+0000) 2024-02-19T20:14:39.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:39.026+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:39.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:39.026+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:39.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:39.026+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:39.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:39.027+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:39.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:39.291+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.292470+0000) 2024-02-19T20:14:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:39.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:39.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:39.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:39.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:39.259+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:39.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:39.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:39.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:39.225+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:39.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:39.225+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:39.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:39.554+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:39.555773+0000) 2024-02-19T20:14:40.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:40.084+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:40.085995+0000) 2024-02-19T20:14:40.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:40.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:40.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:40.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:40.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:40.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:40.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:40.190+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:40.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:40.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:40.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:40.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:40.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:40.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:40.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:40.012+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:40.293+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:40.294448+0000) 2024-02-19T20:14:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:40.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:40.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:40.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:40.223+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:40.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:40.558+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:40.559707+0000) 2024-02-19T20:14:41.211 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:41.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:41.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:41.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:41.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:41.167+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:41.212 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:41.168+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:41.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:41.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:41.213 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:41.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:41.213 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:41.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:41.213 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:41.036+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:41.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:41.085+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.086323+0000) 2024-02-19T20:14:41.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:41.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:41.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:41.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:41.214 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:41.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:41.293+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.294709+0000) 2024-02-19T20:14:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:41.210+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:41.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:41.559+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.560125+0000) 2024-02-19T20:14:42.143 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:41.867+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.868302+0000) 2024-02-19T20:14:42.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:41.868+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.869864+0000) 2024-02-19T20:14:42.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:42.085+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:42.086603+0000) 2024-02-19T20:14:42.144 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:41.870+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57642 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:42.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:41.867+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:41.868454+0000) 2024-02-19T20:14:42.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:42.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:42.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:42.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:42.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:42.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:42.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:42.071+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:42.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:42.293+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:42.294944+0000) 2024-02-19T20:14:42.427 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:42.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:42.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:42.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:42.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:42.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:42.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:42.185+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:42.428 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:42.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:42.428 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:42.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:42.428 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:42.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:42.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:42.143+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:42.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:42.559+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:42.560408+0000) 2024-02-19T20:14:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:43.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:43.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:43.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:43.112+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:43.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:43.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:43.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:43.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:43.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:43.122+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:43.089+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:43.090233+0000) 2024-02-19T20:14:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:43.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:43.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:43.210+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:43.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:43.211+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:43.562 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:43.295+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:43.296986+0000) 2024-02-19T20:14:43.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:43.561+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:43.562603+0000) 2024-02-19T20:14:44.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:44.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:44.235 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:44.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:44.236 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:44.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:44.236 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:44.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:44.236 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:44.091+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.092645+0000) 2024-02-19T20:14:44.236 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:44.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:44.236 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:44.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:44.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:44.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:44.237 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:44.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:44.237 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:44.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:44.237 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:44.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:44.237 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:44.147+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:44.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:44.297+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.298911+0000) 2024-02-19T20:14:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:44.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:44.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:44.564+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:44.565247+0000) 2024-02-19T20:14:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:45.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:45.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:45.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:45.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:45.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:45.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:45.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:45.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:45.145+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:45.095+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.096274+0000) 2024-02-19T20:14:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:45.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:45.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:45.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:45.195+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:45.569 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:45.300+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.301486+0000) 2024-02-19T20:14:45.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:45.567+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:45.568907+0000) 2024-02-19T20:14:46.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:46.095+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.096671+0000) 2024-02-19T20:14:46.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:46.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:46.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:46.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:46.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:46.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:46.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:46.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:46.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:46.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:46.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:46.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:46.209 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:46.129+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:46.210 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:46.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:46.210 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:46.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:46.210 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:46.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:46.210 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:46.097+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:46.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:46.300+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.301841+0000) 2024-02-19T20:14:46.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:46.207+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:46.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:46.568+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.569278+0000) 2024-02-19T20:14:47.144 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:46.874+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.875226+0000) 2024-02-19T20:14:47.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:46.877+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57680 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:47.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:46.874+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.875325+0000) 2024-02-19T20:14:47.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:47.095+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.096881+0000) 2024-02-19T20:14:47.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:46.874+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:46.875591+0000) 2024-02-19T20:14:47.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:47.074+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:47.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:47.074+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:47.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:47.074+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:47.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:47.075+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:47.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:47.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:47.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:47.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:47.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:47.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:47.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:47.189+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:47.431 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:47.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:47.431 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:47.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:47.431 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:47.143+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:47.431 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:47.144+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:47.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:47.301+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.302027+0000) 2024-02-19T20:14:47.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:47.568+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:47.569460+0000) 2024-02-19T20:14:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:48.098+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.099066+0000) 2024-02-19T20:14:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:48.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:48.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:48.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:48.239+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:48.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:48.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:48.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:48.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:48.062+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:48.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:48.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:48.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:48.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:48.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:48.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:48.100+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:48.572 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:48.303+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.304428+0000) 2024-02-19T20:14:48.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:48.571+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:48.572065+0000) 2024-02-19T20:14:49.223 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:49.100+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.101228+0000) 2024-02-19T20:14:49.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:49.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:49.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:49.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:49.224 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:49.085+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:49.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:49.085+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:49.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:49.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:49.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:49.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:49.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:49.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:49.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:49.145+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:49.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:49.305+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.306551+0000) 2024-02-19T20:14:49.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:49.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:49.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:49.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:49.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:49.222+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:49.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:49.223+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:49.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:49.573+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:49.574161+0000) 2024-02-19T20:14:50.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:50.102+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.103383+0000) 2024-02-19T20:14:50.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:50.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:50.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:50.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:50.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:50.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:50.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:50.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:50.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:50.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:50.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:50.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:50.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:50.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:50.204 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:50.190+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:50.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:50.307+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.308625+0000) 2024-02-19T20:14:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:50.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:50.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:50.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:50.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:50.222+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:50.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:50.576+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:50.577842+0000) 2024-02-19T20:14:51.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:51.097+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:51.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:51.097+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:51.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:51.098+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:51.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:51.099+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:51.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:51.102+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.103672+0000) 2024-02-19T20:14:51.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:51.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:51.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:51.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:51.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:51.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:51.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:51.196+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:51.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:51.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:51.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:51.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:51.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:51.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:51.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:51.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:51.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:51.307+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.308941+0000) 2024-02-19T20:14:51.880 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:51.578+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.578214+0000) 2024-02-19T20:14:52.149 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:51.885+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57721 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:52.150 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:51.883+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.883919+0000) 2024-02-19T20:14:52.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:51.882+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.882897+0000) 2024-02-19T20:14:52.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:52.103+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.103864+0000) 2024-02-19T20:14:52.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:51.882+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:51.883014+0000) 2024-02-19T20:14:52.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:52.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:52.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:52.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:52.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:52.149+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:52.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:52.150+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:52.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:52.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:52.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:52.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:52.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:52.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:52.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:52.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:52.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:52.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:52.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:52.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:52.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:52.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:52.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:52.191+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:52.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:52.309+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.309138+0000) 2024-02-19T20:14:52.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:52.578+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:52.578376+0000) 2024-02-19T20:14:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:53.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:53.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:53.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:53.159+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:53.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:53.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:53.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:53.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:53.104+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.105935+0000) 2024-02-19T20:14:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:53.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:53.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:53.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:53.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:53.165+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:53.580 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:53.310+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.311739+0000) 2024-02-19T20:14:53.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:53.579+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:53.580445+0000) 2024-02-19T20:14:54.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:54.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:54.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:54.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:54.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:54.112+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:54.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:54.113+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:54.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:54.107+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.108468+0000) 2024-02-19T20:14:54.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:54.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:54.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:54.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:54.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:54.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:54.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:54.132+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:54.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:54.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:54.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:54.256+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:54.312+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.313758+0000) 2024-02-19T20:14:54.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:54.581+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:54.582623+0000) 2024-02-19T20:14:55.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:55.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:55.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:55.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:55.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:55.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:55.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:55.096+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:55.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:55.110+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.111141+0000) 2024-02-19T20:14:55.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:55.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:55.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:55.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:55.223 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:55.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:55.223 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:55.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:55.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:55.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:55.220+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:55.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:55.314+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.315802+0000) 2024-02-19T20:14:55.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:55.585+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:55.586218+0000) 2024-02-19T20:14:56.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:56.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:56.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:56.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:56.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:56.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:56.257 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:56.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:56.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:56.110+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.111631+0000) 2024-02-19T20:14:56.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:56.127+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:56.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:56.127+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:56.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:56.127+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:56.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:56.127+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:56.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:56.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:56.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:56.256+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:56.315+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.316175+0000) 2024-02-19T20:14:56.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:56.585+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.586697+0000) 2024-02-19T20:14:57.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:14:56.892+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57759 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:14:57.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:56.889+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.890707+0000) 2024-02-19T20:14:57.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:56.889+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.890388+0000) 2024-02-19T20:14:57.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:57.110+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.111885+0000) 2024-02-19T20:14:57.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:56.888+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:56.889954+0000) 2024-02-19T20:14:57.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:57.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:57.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:57.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:57.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:57.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:57.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:57.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:57.438 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:57.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:57.438 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:57.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:57.438 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:57.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:57.438 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:57.177+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:57.438 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:57.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:57.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:57.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:57.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:57.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:57.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:57.229+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:57.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:57.315+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.316429+0000) 2024-02-19T20:14:57.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:57.585+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:57.586877+0000) 2024-02-19T20:14:58.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:58.113+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.114408+0000) 2024-02-19T20:14:58.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:58.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:58.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:58.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:58.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:58.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:58.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:58.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:58.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:58.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:58.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:58.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:58.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:58.045+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:58.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:58.046+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:58.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:58.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:58.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:58.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:58.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:58.277+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:58.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:58.278+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:58.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:58.318+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.319145+0000) 2024-02-19T20:14:58.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:58.588+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:58.589384+0000) 2024-02-19T20:14:59.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:59.116+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.117158+0000) 2024-02-19T20:14:59.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:59.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:59.206 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:59.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:59.206 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:59.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:14:59.206 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:59.074+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:59.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:59.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:59.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:14:59.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:14:59.201+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:14:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:59.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:59.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:59.292+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:14:59.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:59.293+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:14:59.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:14:59.320+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.321618+0000) 2024-02-19T20:14:59.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:14:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:14:59.590+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:14:59.591885+0000) 2024-02-19T20:15:00.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:00.118+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.119180+0000) 2024-02-19T20:15:00.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:00.082+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:00.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:00.082+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:00.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:00.082+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:00.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:00.083+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:00.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:00.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:00.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:00.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:00.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:00.223+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:00.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:00.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:00.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:00.278+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:00.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:00.279+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:00.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:00.322+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.323708+0000) 2024-02-19T20:15:00.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:00.593+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:00.594050+0000) 2024-02-19T20:15:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:01.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:01.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:01.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:01.104+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:01.118+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.119485+0000) 2024-02-19T20:15:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:01.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:01.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:01.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:01.192+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:01.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:01.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:01.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:01.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:01.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:01.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:01.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:01.304+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:01.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:01.323+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.324138+0000) 2024-02-19T20:15:01.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:01.593+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.594477+0000) 2024-02-19T20:15:02.152 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:01.899+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57801 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:02.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:01.897+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.898070+0000) 2024-02-19T20:15:02.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:01.897+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.898096+0000) 2024-02-19T20:15:02.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:02.118+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.119720+0000) 2024-02-19T20:15:02.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:01.900+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:01.901496+0000) 2024-02-19T20:15:02.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:02.071+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:02.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:02.071+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:02.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:02.071+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:02.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:02.071+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:02.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:02.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:02.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:02.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:02.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:02.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:02.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:02.181+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:02.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:02.323+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.324391+0000) 2024-02-19T20:15:02.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:02.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:02.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:02.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:02.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:02.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:02.443 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:02.348+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:02.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:02.593+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:02.594695+0000) 2024-02-19T20:15:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:03.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:03.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:03.121+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:03.122+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:03.122+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.123232+0000) 2024-02-19T20:15:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:03.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:03.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:03.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:03.169+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:03.597 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:03.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:03.597 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:03.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:03.597 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:03.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:03.597 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:03.325+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.326464+0000) 2024-02-19T20:15:03.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:03.325+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:03.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:03.595+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:03.596809+0000) 2024-02-19T20:15:04.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:04.124+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.125373+0000) 2024-02-19T20:15:04.208 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:04.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:04.208 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:04.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:04.208 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:04.080+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:04.209 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:04.081+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:04.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:04.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:04.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:04.206+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:04.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:04.207+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:04.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:04.310+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:04.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:04.310+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:04.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:04.310+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:04.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:04.311+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:04.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:04.329+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.330487+0000) 2024-02-19T20:15:04.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:04.598+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:04.599064+0000) 2024-02-19T20:15:05.274 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:05.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:05.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:05.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:05.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:05.126+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:05.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:05.127+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:05.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:05.128+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.129016+0000) 2024-02-19T20:15:05.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:05.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:05.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:05.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:05.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:05.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:05.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:05.172+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:05.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:05.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:05.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:05.273+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:05.274+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:05.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:05.331+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.332567+0000) 2024-02-19T20:15:05.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:05.600+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:05.601045+0000) 2024-02-19T20:15:06.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:06.128+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.129444+0000) 2024-02-19T20:15:06.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:06.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:06.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:06.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:06.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:06.193+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:06.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:06.194+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:06.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:06.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:06.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:06.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:06.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:06.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:06.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:06.110+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:06.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:06.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:06.281+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:06.282+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:06.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:06.331+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.332916+0000) 2024-02-19T20:15:06.901 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:06.600+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.601481+0000) 2024-02-19T20:15:07.153 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:06.906+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57839 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:07.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:06.905+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.906162+0000) 2024-02-19T20:15:07.154 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:06.904+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.905104+0000) 2024-02-19T20:15:07.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:06.903+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:06.904406+0000) 2024-02-19T20:15:07.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:07.128+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.129608+0000) 2024-02-19T20:15:07.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:07.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:07.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:07.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:07.447 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:07.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:07.447 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:07.153+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:07.447 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:07.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:07.447 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:07.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:07.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:07.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:07.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:07.189+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:07.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:07.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:07.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:07.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:07.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:07.302+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:07.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:07.303+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:07.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:07.332+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.333116+0000) 2024-02-19T20:15:07.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:07.600+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:07.601677+0000) 2024-02-19T20:15:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:08.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:08.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:08.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:08.177+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:08.132+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.133058+0000) 2024-02-19T20:15:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:08.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:08.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:08.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:08.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:08.163+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:08.604 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:08.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:08.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:08.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:08.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:08.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:08.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:08.330+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:08.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:08.334+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.335038+0000) 2024-02-19T20:15:08.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:08.602+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:08.603776+0000) 2024-02-19T20:15:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:09.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:09.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:09.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:09.132+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:09.134+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.135089+0000) 2024-02-19T20:15:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:09.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:09.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:09.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:09.136+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:09.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:09.337+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.338457+0000) 2024-02-19T20:15:09.606 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:09.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:09.607 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:09.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:09.607 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:09.380+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:09.607 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:09.381+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:09.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:09.605+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:09.606301+0000) 2024-02-19T20:15:10.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:10.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:10.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:10.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:10.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:10.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:10.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:10.110+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:10.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:10.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:10.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:10.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:10.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:10.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:10.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:10.082+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:10.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:10.136+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.137100+0000) 2024-02-19T20:15:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:10.341+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.342141+0000) 2024-02-19T20:15:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:10.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:10.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:10.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:10.389+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:10.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:10.607+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:10.608462+0000) 2024-02-19T20:15:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:11.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:11.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:11.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:11.082+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:11.136+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.137449+0000) 2024-02-19T20:15:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:11.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:11.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:11.066+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:11.067+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:11.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:11.341+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.342558+0000) 2024-02-19T20:15:11.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:11.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:11.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:11.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:11.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:11.376+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:11.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:11.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:11.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:11.607+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.608788+0000) 2024-02-19T20:15:12.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:11.914+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57880 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:11.911+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.912205+0000) 2024-02-19T20:15:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:11.911+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.912575+0000) 2024-02-19T20:15:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:12.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:12.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:12.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:12.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:12.089+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:12.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:11.910+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:11.911161+0000) 2024-02-19T20:15:12.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:12.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:12.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:12.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:12.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:12.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:12.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:12.071+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:12.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:12.136+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.137640+0000) 2024-02-19T20:15:12.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:12.341+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.342747+0000) 2024-02-19T20:15:12.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:12.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:12.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:12.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:12.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:12.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:12.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:12.347+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:12.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:12.608+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:12.609033+0000) 2024-02-19T20:15:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:13.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:13.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:13.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:13.107+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:13.138+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.139704+0000) 2024-02-19T20:15:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:13.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:13.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:13.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:13.097+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:13.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:13.343+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.344802+0000) 2024-02-19T20:15:13.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:13.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:13.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:13.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:13.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:13.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:13.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:13.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:13.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:13.610+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:13.611133+0000) 2024-02-19T20:15:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:14.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:14.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:14.097+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:14.097+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:14.142+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.143326+0000) 2024-02-19T20:15:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:14.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:14.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:14.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:14.052+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:14.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:14.345+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.346977+0000) 2024-02-19T20:15:14.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:14.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:14.614 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:14.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:14.614 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:14.418+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:14.614 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:14.419+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:14.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:14.612+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:14.613357+0000) 2024-02-19T20:15:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:15.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:15.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:15.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:15.088+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:15.144+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.145369+0000) 2024-02-19T20:15:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:15.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:15.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:15.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:15.005+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:15.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:15.348+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.349601+0000) 2024-02-19T20:15:15.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:15.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:15.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:15.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:15.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:15.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:15.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:15.388+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:15.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:15.615+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:15.616063+0000) 2024-02-19T20:15:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:16.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:16.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:16.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:16.099+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:16.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:16.144+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.145722+0000) 2024-02-19T20:15:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:16.040+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:16.040+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:16.040+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:16.040+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:16.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:16.349+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.350089+0000) 2024-02-19T20:15:16.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:16.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:16.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:16.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:16.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:16.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:16.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:16.422+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:16.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:16.615+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.616511+0000) 2024-02-19T20:15:17.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:16.921+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57918 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:16.918+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.919531+0000) 2024-02-19T20:15:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:16.919+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.920088+0000) 2024-02-19T20:15:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:17.093+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:17.093+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:17.093+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:17.094+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:17.144+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.145898+0000) 2024-02-19T20:15:17.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:16.918+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:16.919356+0000) 2024-02-19T20:15:17.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:17.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:17.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:17.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:17.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:17.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:17.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:17.020+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:17.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:17.349+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.350286+0000) 2024-02-19T20:15:17.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:17.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:17.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:17.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:17.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:17.460+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:17.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:17.461+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:17.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:17.615+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:17.616773+0000) 2024-02-19T20:15:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:18.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:18.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:18.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:18.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:18.040+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:18.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:18.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:18.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:18.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:18.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:18.144+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:18.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:18.148+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.149172+0000) 2024-02-19T20:15:18.619 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:18.351+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.352289+0000) 2024-02-19T20:15:18.620 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:18.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:18.620 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:18.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:18.620 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:18.425+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:18.621 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:18.426+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:18.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:18.618+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:18.619307+0000) 2024-02-19T20:15:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:19.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:19.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:19.019+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:19.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:19.020+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:19.150+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.151573+0000) 2024-02-19T20:15:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:19.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:19.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:19.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:19.192+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:19.622 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:19.353+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.354504+0000) 2024-02-19T20:15:19.622 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:19.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:19.622 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:19.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:19.622 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:19.409+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:19.622 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:19.410+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:19.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:19.620+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:19.621812+0000) 2024-02-19T20:15:20.203 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:20.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:20.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:20.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:20.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:20.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:20.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:20.059+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:20.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:20.153+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.154236+0000) 2024-02-19T20:15:20.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:20.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:20.204 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:20.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:20.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:20.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:20.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:20.198+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:20.457 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:20.356+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.357197+0000) 2024-02-19T20:15:20.457 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:20.444+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:20.457 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:20.444+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:20.458 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:20.444+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:20.458 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:20.445+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:20.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:20.623+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:20.624261+0000) 2024-02-19T20:15:21.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:21.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:21.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:21.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:21.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:21.109+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:21.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:21.110+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:21.149+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:21.149+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:21.149+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:21.149+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:21.153+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.154719+0000) 2024-02-19T20:15:21.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:21.356+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.357535+0000) 2024-02-19T20:15:21.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:21.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:21.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:21.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:21.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:21.417+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:21.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:21.418+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:21.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:21.623+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.624750+0000) 2024-02-19T20:15:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:21.928+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57959 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:21.927+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.928238+0000) 2024-02-19T20:15:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:21.925+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.926161+0000) 2024-02-19T20:15:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:22.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:22.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:22.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:22.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:22.088+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:22.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:21.926+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:21.927597+0000) 2024-02-19T20:15:22.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:22.107+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:22.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:22.107+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:22.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:22.107+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:22.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:22.108+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:22.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:22.153+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.154889+0000) 2024-02-19T20:15:22.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:22.356+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.357797+0000) 2024-02-19T20:15:22.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:22.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:22.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:22.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:22.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:22.434+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:22.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:22.435+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:22.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:22.624+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:22.625041+0000) 2024-02-19T20:15:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:23.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:23.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:23.121+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:23.122+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:23.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:23.156+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.157708+0000) 2024-02-19T20:15:23.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:23.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:23.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:23.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:23.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:23.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:23.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:23.091+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:23.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:23.360+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.361345+0000) 2024-02-19T20:15:23.628 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:23.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:23.628 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:23.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:23.628 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:23.386+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:23.628 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:23.386+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:23.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:23.626+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:23.627143+0000) 2024-02-19T20:15:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:24.138+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:24.138+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:24.138+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:24.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:24.139+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:24.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:24.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:24.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:24.087+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:24.158+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.159904+0000) 2024-02-19T20:15:24.629 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:24.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:24.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:24.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:24.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:24.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:24.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:24.351+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:24.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:24.363+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.364925+0000) 2024-02-19T20:15:24.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:24.628+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:24.629533+0000) 2024-02-19T20:15:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:25.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:25.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:25.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:25.066+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:25.160+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.161966+0000) 2024-02-19T20:15:25.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:25.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:25.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:25.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:25.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:25.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:25.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:25.097+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:25.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:25.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:25.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:25.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:25.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:25.363+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:25.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:25.364+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:25.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:25.366+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.367643+0000) 2024-02-19T20:15:25.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:25.630+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:25.631919+0000) 2024-02-19T20:15:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:26.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:26.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:26.120+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:26.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:26.121+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:26.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:26.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:26.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:26.086+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:26.161+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.162339+0000) 2024-02-19T20:15:26.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:26.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:26.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:26.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:26.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:26.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:26.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:26.316+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:26.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:26.367+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.368124+0000) 2024-02-19T20:15:26.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:26.631+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.632297+0000) 2024-02-19T20:15:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:26.933+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.934377+0000) 2024-02-19T20:15:27.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:26.935+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 57997 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:26.933+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.934432+0000) 2024-02-19T20:15:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:27.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:27.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:27.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:27.135+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:27.161+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.162543+0000) 2024-02-19T20:15:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:26.932+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:26.933278+0000) 2024-02-19T20:15:27.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:27.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:27.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:27.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:27.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:27.081+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:27.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:27.082+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:27.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:27.344+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:27.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:27.344+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:27.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:27.344+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:27.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:27.345+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:27.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:27.367+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.368282+0000) 2024-02-19T20:15:27.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:27.631+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:27.632576+0000) 2024-02-19T20:15:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:28.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:28.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:28.039+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:28.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:28.039+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:28.163+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.164461+0000) 2024-02-19T20:15:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:28.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:28.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:28.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:28.169+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:28.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:28.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:28.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:28.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:28.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:28.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:28.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:28.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:28.635 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:28.369+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.370297+0000) 2024-02-19T20:15:28.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:28.633+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:28.634700+0000) 2024-02-19T20:15:29.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:29.165+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.166576+0000) 2024-02-19T20:15:29.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:29.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:29.215 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:29.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:29.216 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:29.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:29.216 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:29.063+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:29.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:29.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:29.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:29.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:29.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:29.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:29.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:29.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:29.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:29.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:29.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:29.346+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:29.347+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:29.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:29.371+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.372394+0000) 2024-02-19T20:15:29.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:29.635+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:29.636853+0000) 2024-02-19T20:15:30.204 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:30.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:30.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:30.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:30.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:30.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:30.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:30.056+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:30.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:30.169+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.170214+0000) 2024-02-19T20:15:30.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:30.183+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:30.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:30.183+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:30.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:30.183+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:30.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:30.183+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:30.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:30.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:30.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:30.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:30.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:30.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:30.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:30.369+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:30.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:30.374+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.375052+0000) 2024-02-19T20:15:30.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:30.638+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:30.639466+0000) 2024-02-19T20:15:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:31.082+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:31.082+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:31.082+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:31.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:31.082+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:31.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:31.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:31.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:31.140+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:31.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:31.169+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.170577+0000) 2024-02-19T20:15:31.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:31.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:31.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:31.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:31.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:31.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:31.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:31.344+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:31.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:31.374+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.375435+0000) 2024-02-19T20:15:31.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:31.638+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.639860+0000) 2024-02-19T20:15:32.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:31.943+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58038 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:32.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:31.940+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.941845+0000) 2024-02-19T20:15:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:31.939+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.940775+0000) 2024-02-19T20:15:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:32.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:32.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:32.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:32.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:32.090+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:31.940+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:31.941149+0000) 2024-02-19T20:15:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:32.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:32.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:32.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:32.091+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:32.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:32.169+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.170774+0000) 2024-02-19T20:15:32.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:32.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:32.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:32.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:32.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:32.343+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:32.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:32.344+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:32.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:32.374+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.375615+0000) 2024-02-19T20:15:32.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:32.639+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:32.640014+0000) 2024-02-19T20:15:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:33.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:33.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:33.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:33.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:33.087+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:33.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:33.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:33.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:33.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:33.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:33.141+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:33.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:33.142+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:33.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:33.173+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.174424+0000) 2024-02-19T20:15:33.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:33.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:33.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:33.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:33.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:33.329+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:33.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:33.330+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:33.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:33.376+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.377679+0000) 2024-02-19T20:15:33.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:33.641+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:33.642301+0000) 2024-02-19T20:15:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:34.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:34.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:34.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:34.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:34.058+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:34.175+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.176454+0000) 2024-02-19T20:15:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:34.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:34.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:34.186+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:34.186+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:34.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:34.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:34.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:34.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:34.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:34.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:34.645 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:34.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:34.646 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:34.378+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.379742+0000) 2024-02-19T20:15:34.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:34.644+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:34.645080+0000) 2024-02-19T20:15:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:35.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:35.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:35.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:35.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:35.105+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:35.177+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.178925+0000) 2024-02-19T20:15:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:35.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:35.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:35.190+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:35.191+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:35.647 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:35.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:35.647 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:35.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:35.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:35.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:35.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:35.295+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:35.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:35.380+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.381856+0000) 2024-02-19T20:15:35.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:35.646+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:35.647366+0000) 2024-02-19T20:15:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:36.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:36.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:36.143+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:36.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:36.144+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:36.178+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.179416+0000) 2024-02-19T20:15:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:36.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:36.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:36.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:36.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:36.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:36.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:36.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:36.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:36.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:36.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:36.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:36.298+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:36.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:36.381+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.382237+0000) 2024-02-19T20:15:36.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:36.646+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.647882+0000) 2024-02-19T20:15:37.250 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:36.950+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58076 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:37.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:36.947+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.948310+0000) 2024-02-19T20:15:37.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:37.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:37.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:37.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:37.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:37.249+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:37.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:36.947+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.948439+0000) 2024-02-19T20:15:37.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:37.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:37.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:37.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:37.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:37.099+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:37.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:37.099+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:37.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:36.948+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:36.949922+0000) 2024-02-19T20:15:37.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:37.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:37.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:37.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:37.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:37.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:37.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:37.163+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:37.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:37.178+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.179586+0000) 2024-02-19T20:15:37.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:37.249+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:37.382+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.382437+0000) 2024-02-19T20:15:37.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:37.648+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:37.648404+0000) 2024-02-19T20:15:38.232 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:38.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:38.233 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:38.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:38.233 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:38.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:38.233 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:38.095+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:38.233 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:38.182+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.182241+0000) 2024-02-19T20:15:38.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:38.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:38.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:38.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:38.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:38.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:38.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:38.189+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:38.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:38.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:38.232+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:38.233+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:38.384+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.384473+0000) 2024-02-19T20:15:38.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:38.652+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:38.652114+0000) 2024-02-19T20:15:39.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:39.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:39.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:39.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:39.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:39.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:39.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:39.062+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:39.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:39.183+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.184264+0000) 2024-02-19T20:15:39.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:39.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:39.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:39.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:39.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:39.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:39.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:39.199+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:39.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:39.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:39.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:39.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:39.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:39.275+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:39.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:39.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:39.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:39.386+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.387128+0000) 2024-02-19T20:15:39.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:39.654+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:39.655917+0000) 2024-02-19T20:15:40.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:40.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:40.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:40.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:40.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:40.052+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:40.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:40.053+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:40.253 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:40.185+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.186260+0000) 2024-02-19T20:15:40.253 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:40.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:40.254 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:40.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:40.254 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:40.189+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:40.254 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:40.190+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:40.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:40.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:40.251+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:40.252+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:40.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:40.388+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.389641+0000) 2024-02-19T20:15:40.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:40.657+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:40.658195+0000) 2024-02-19T20:15:41.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:41.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:41.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:41.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.006+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:41.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:41.185+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.186650+0000) 2024-02-19T20:15:41.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:41.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:41.223 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:41.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:41.223 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:41.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:41.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:41.221+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:41.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:41.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:41.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:41.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:41.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:41.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:41.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:41.240+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:41.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:41.389+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.390082+0000) 2024-02-19T20:15:41.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.657+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.658558+0000) 2024-02-19T20:15:42.011 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:41.957+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58117 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:42.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:41.955+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.956383+0000) 2024-02-19T20:15:42.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:41.956+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.957139+0000) 2024-02-19T20:15:42.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.954+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:41.955844+0000) 2024-02-19T20:15:42.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:42.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:42.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:42.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:41.981+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:42.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:42.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:42.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:42.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:42.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:42.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:42.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:42.195+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:42.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:42.185+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.186816+0000) 2024-02-19T20:15:42.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:42.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:42.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:42.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:42.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:42.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:42.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:42.251+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:42.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:42.389+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.390342+0000) 2024-02-19T20:15:43.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:42.657+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:42.658761+0000) 2024-02-19T20:15:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:43.187+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.188785+0000) 2024-02-19T20:15:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:43.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.007+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:43.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.008+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:43.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:43.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:43.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:43.189+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:43.661 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:43.392+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.393758+0000) 2024-02-19T20:15:43.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:43.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:43.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:43.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:43.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:43.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:43.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:43.293+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:43.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.659+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:43.660748+0000) 2024-02-19T20:15:43.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:43.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:43.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:44.230 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:43.972+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:44.230 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:44.190+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.191262+0000) 2024-02-19T20:15:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:44.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:44.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:44.301+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:44.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:44.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:44.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:44.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:44.229+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:44.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:44.396+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.397312+0000) 2024-02-19T20:15:45.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:44.661+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:44.662934+0000) 2024-02-19T20:15:45.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:45.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:45.192+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.193738+0000) 2024-02-19T20:15:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:45.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.005+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:45.185+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:45.185+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:45.185+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:45.186+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:45.665 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:45.398+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.399543+0000) 2024-02-19T20:15:45.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:45.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:45.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:45.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:45.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:45.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:45.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:45.329+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:45.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.664+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:45.665408+0000) 2024-02-19T20:15:45.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:45.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:45.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.983+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:46.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:45.984+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:46.193+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.194143+0000) 2024-02-19T20:15:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:46.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:46.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:46.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:46.163+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:46.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:46.399+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.400063+0000) 2024-02-19T20:15:46.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:46.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:46.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:46.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:46.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:46.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:46.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:46.303+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:46.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:46.664+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.665940+0000) 2024-02-19T20:15:47.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:46.964+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58156 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:47.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:46.963+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.964176+0000) 2024-02-19T20:15:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:47.193+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.194316+0000) 2024-02-19T20:15:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:46.944+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:46.944+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:46.944+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:46.945+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:46.962+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.963235+0000) 2024-02-19T20:15:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:46.961+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:46.963014+0000) 2024-02-19T20:15:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:47.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:47.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:47.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:47.125+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:47.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:47.399+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.400239+0000) 2024-02-19T20:15:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:47.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:47.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:47.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:47.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:47.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:47.290+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:47.919 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:47.665+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:47.666128+0000) 2024-02-19T20:15:47.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:47.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:47.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:47.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:47.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:47.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:48.198 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:47.919+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:48.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:48.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:48.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:48.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:48.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:48.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:48.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:48.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:48.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:48.401+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.402632+0000) 2024-02-19T20:15:48.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:48.196+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.197757+0000) 2024-02-19T20:15:48.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:48.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:48.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:48.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:48.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:48.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:48.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:48.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:48.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:48.668+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:48.669682+0000) 2024-02-19T20:15:49.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:48.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:49.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:48.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:49.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:48.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:49.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:48.954+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:49.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:49.199+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.200158+0000) 2024-02-19T20:15:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:49.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:49.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:49.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:49.071+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:49.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:49.403+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.404715+0000) 2024-02-19T20:15:49.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:49.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:49.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:49.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:49.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:49.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:49.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:49.363+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:49.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:49.672+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:49.673311+0000) 2024-02-19T20:15:49.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:49.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:49.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:49.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:49.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:49.948+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:50.202 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:49.949+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:50.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:50.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:50.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:50.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:50.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:50.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:50.203 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:50.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:50.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:50.405+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.406772+0000) 2024-02-19T20:15:50.480 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:50.201+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.202194+0000) 2024-02-19T20:15:50.480 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:50.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:50.480 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:50.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:50.481 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:50.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:50.481 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:50.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:50.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:50.675+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:50.676938+0000) 2024-02-19T20:15:51.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:50.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:51.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:50.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:51.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:50.953+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:51.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:50.954+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:51.201+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.202779+0000) 2024-02-19T20:15:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:51.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:51.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:51.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:51.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:51.677 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:51.406+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.407204+0000) 2024-02-19T20:15:51.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:51.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:51.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:51.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:51.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:51.374+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:51.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:51.374+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:51.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:51.676+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.677466+0000) 2024-02-19T20:15:51.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:51.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:51.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:51.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:52.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:51.972+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58196 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:51.968+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.969838+0000) 2024-02-19T20:15:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:52.201+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.203000+0000) 2024-02-19T20:15:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:51.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:51.953+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:52.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:51.969+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.970242+0000) 2024-02-19T20:15:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:51.969+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:51.970562+0000) 2024-02-19T20:15:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:52.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:52.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:52.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:52.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:52.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:52.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:52.677 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:52.406+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.407402+0000) 2024-02-19T20:15:52.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:52.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:52.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:52.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:52.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:52.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:52.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:52.330+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:52.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:52.676+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:52.677713+0000) 2024-02-19T20:15:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:53.204+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.205057+0000) 2024-02-19T20:15:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:52.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:52.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:52.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:53.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:52.966+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:53.680 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.408+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.409638+0000) 2024-02-19T20:15:53.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:53.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:53.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:53.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:53.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:53.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:53.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:53.350+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:53.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:53.679+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:53.680182+0000) 2024-02-19T20:15:53.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:53.932+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:53.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:53.932+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:54.207 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:53.932+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:54.208 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:53.933+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:54.208 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:54.208 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:54.208 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:54.208 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:53.990+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:54.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:54.412+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.413213+0000) 2024-02-19T20:15:54.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:54.206+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.207591+0000) 2024-02-19T20:15:54.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:54.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:54.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:54.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:54.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:54.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:54.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:54.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:54.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:54.681+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:54.682980+0000) 2024-02-19T20:15:55.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:54.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:55.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:54.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:55.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:54.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:55.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:54.976+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:55.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:54.970+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:55.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:54.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:55.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:54.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:55.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:54.971+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:55.210+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.211224+0000) 2024-02-19T20:15:55.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:55.415+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.416854+0000) 2024-02-19T20:15:55.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:55.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:55.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:55.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:55.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:55.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:55.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:55.306+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:55.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:55.684+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:55.685511+0000) 2024-02-19T20:15:56.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:55.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:56.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:55.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:56.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:55.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:56.213 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:55.954+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:56.213 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:55.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:56.213 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:55.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:56.213 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:55.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:56.214 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:55.982+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:56.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:56.416+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.417328+0000) 2024-02-19T20:15:56.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:56.210+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.211618+0000) 2024-02-19T20:15:56.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:56.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:56.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:56.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:56.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:56.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:56.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:56.339+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:56.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:56.685+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.686313+0000) 2024-02-19T20:15:56.998 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:15:56.979+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58235 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:15:56.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:56.976+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.977993+0000) 2024-02-19T20:15:56.999 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:56.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:56.999 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:56.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:57.000 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:56.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:57.000 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:56.915+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:57.000 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:56.976+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.977220+0000) 2024-02-19T20:15:57.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:56.976+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:56.977381+0000) 2024-02-19T20:15:57.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:56.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:57.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:56.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:57.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:56.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:57.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:56.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:57.210+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.211848+0000) 2024-02-19T20:15:57.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:57.416+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.417526+0000) 2024-02-19T20:15:57.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:57.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:57.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:57.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:57.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:57.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:57.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:57.387+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:58.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:57.685+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:57.686482+0000) 2024-02-19T20:15:58.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:57.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:58.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:57.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:58.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:57.998+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:58.000 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:57.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:58.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:57.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:58.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:57.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:58.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:57.926+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:58.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:57.999+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:58.213+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.214065+0000) 2024-02-19T20:15:58.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:58.418+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.419984+0000) 2024-02-19T20:15:58.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:58.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:58.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:58.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:58.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:58.420+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:58.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:58.421+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:58.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:58.687+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:58.688496+0000) 2024-02-19T20:15:59.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:58.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:59.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:58.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:59.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:58.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:15:59.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:58.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:15:59.218 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:58.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:59.219 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:58.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:59.219 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:58.958+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:15:59.219 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:58.959+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:59.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:59.421+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.422625+0000) 2024-02-19T20:15:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:59.215+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.216113+0000) 2024-02-19T20:15:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:59.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:59.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:59.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:59.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:15:59.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:15:59.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:15:59.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:15:59.689+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:15:59.690481+0000) 2024-02-19T20:16:00.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:59.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:00.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:59.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:00.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:59.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:00.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:15:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:15:59.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:00.217+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.218251+0000) 2024-02-19T20:16:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:00.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:00.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:00.005+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:00.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:00.006+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:00.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:00.423+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.424895+0000) 2024-02-19T20:16:00.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:00.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:00.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:00.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:00.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:00.465+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:00.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:00.466+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:01.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:00.691+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:00.692610+0000) 2024-02-19T20:16:01.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:00.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:01.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:00.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:01.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:00.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:01.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:00.891+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:01.217+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.218688+0000) 2024-02-19T20:16:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.037+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.037+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.037+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.038+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:01.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:01.424+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.425497+0000) 2024-02-19T20:16:01.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:01.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:01.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:01.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:01.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:01.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:01.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:01.425+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:01.982 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.692+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.693068+0000) 2024-02-19T20:16:01.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:01.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:01.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:01.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:01.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:01.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:01.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:01.921+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:02.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:01.987+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58274 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:01.983+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.984606+0000) 2024-02-19T20:16:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:01.984+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.985704+0000) 2024-02-19T20:16:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:02.217+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.218904+0000) 2024-02-19T20:16:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.983+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:01.984802+0000) 2024-02-19T20:16:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.989+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:01.990+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:02.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:02.424+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.425707+0000) 2024-02-19T20:16:02.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:02.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:02.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:02.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:02.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:02.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:02.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:02.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:02.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:02.692+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:02.693233+0000) 2024-02-19T20:16:03.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:02.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:03.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:02.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:03.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:02.986+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:03.221 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:02.987+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:03.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:02.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:03.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:02.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:03.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:02.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:03.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:02.961+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:03.491 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:03.426+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.427791+0000) 2024-02-19T20:16:03.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:03.219+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.220933+0000) 2024-02-19T20:16:03.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:03.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:03.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:03.445+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:03.445+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:03.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:03.694+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:03.695213+0000) 2024-02-19T20:16:04.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:03.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:04.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:03.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:04.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:03.966+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:04.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:03.967+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:04.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:03.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:04.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:03.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:04.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:03.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:04.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:03.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:04.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:04.222+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.223101+0000) 2024-02-19T20:16:04.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:04.430+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.431288+0000) 2024-02-19T20:16:04.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:04.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:04.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:04.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:04.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:04.466+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:04.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:04.466+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:04.980 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:04.696+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:04.697218+0000) 2024-02-19T20:16:04.980 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:04.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:05.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:05.224+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.225211+0000) 2024-02-19T20:16:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:04.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:04.979+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:04.980+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:04.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:04.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:04.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:04.994+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:05.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:05.433+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.434712+0000) 2024-02-19T20:16:05.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:05.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:05.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:05.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:05.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:05.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:05.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:05.487+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:05.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:05.698+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:05.699245+0000) 2024-02-19T20:16:06.225 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:05.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:06.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:05.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:06.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:05.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:06.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:05.935+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:06.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:06.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:06.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:06.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:06.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:06.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:06.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:06.007+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:06.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:06.434+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.435145+0000) 2024-02-19T20:16:06.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:06.224+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.225779+0000) 2024-02-19T20:16:06.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:06.698+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.699667+0000) 2024-02-19T20:16:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:06.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:06.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:06.497+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:06.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:07.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:06.994+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58314 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:06.991+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.992053+0000) 2024-02-19T20:16:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:07.224+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.225964+0000) 2024-02-19T20:16:07.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:06.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:07.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:06.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:07.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:06.982+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:07.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:06.983+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:07.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:06.991+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.992475+0000) 2024-02-19T20:16:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:06.990+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:06.991611+0000) 2024-02-19T20:16:07.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:07.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:07.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:07.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.022+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:07.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.434+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.435324+0000) 2024-02-19T20:16:07.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:07.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:07.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:07.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:07.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:07.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:07.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:07.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:07.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:07.698+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:07.699865+0000) 2024-02-19T20:16:07.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:07.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:07.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:07.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:07.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:07.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:08.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:08.227+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.228083+0000) 2024-02-19T20:16:08.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:07.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:07.994+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:08.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:08.436+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.437519+0000) 2024-02-19T20:16:08.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:08.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:08.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:08.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:08.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:08.470+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:08.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:08.470+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:08.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:08.701+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:08.702464+0000) 2024-02-19T20:16:08.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:08.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:08.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:08.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:08.948 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:08.946+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:09.230 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:08.946+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:09.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:09.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:09.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:09.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:09.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:09.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:09.231 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:09.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:09.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:09.440+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.441042+0000) 2024-02-19T20:16:09.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:09.229+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.230274+0000) 2024-02-19T20:16:09.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:09.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:09.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:09.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:09.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:09.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:09.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:09.423+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:09.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:09.703+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:09.704629+0000) 2024-02-19T20:16:10.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:10.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:10.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:10.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:10.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:10.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:10.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:10.015+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:10.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:09.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:10.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:09.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:10.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:09.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:10.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:09.899+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:10.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:10.231+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.232414+0000) 2024-02-19T20:16:10.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:10.443+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.444599+0000) 2024-02-19T20:16:10.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:10.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:10.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:10.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:10.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:10.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:10.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:10.472+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:11.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:10.705+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:10.706880+0000) 2024-02-19T20:16:11.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:10.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:11.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:10.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:11.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:10.901+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:11.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:10.902+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:11.232+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.233093+0000) 2024-02-19T20:16:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:11.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:11.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:11.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:11.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:11.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:11.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:11.444+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.445192+0000) 2024-02-19T20:16:11.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:11.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:11.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:11.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:11.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:11.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:11.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:11.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:11.996 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:11.706+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.707500+0000) 2024-02-19T20:16:11.996 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:11.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:11.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:11.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:11.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:11.923+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:11.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:11.924+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:12.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:12.001+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58353 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:12.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:11.998+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.999149+0000) 2024-02-19T20:16:12.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:11.997+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:11.998994+0000) 2024-02-19T20:16:12.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:12.232+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.233272+0000) 2024-02-19T20:16:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:11.999+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.000049+0000) 2024-02-19T20:16:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:12.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:12.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:12.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.444+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.445386+0000) 2024-02-19T20:16:12.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:12.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:12.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:12.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:12.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:12.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:12.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:12.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:12.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:12.706+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:12.707675+0000) 2024-02-19T20:16:12.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:12.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:12.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:12.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:12.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:12.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:12.989 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:12.888+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:13.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:13.234+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.236003+0000) 2024-02-19T20:16:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:12.987+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:13.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:13.446+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.447636+0000) 2024-02-19T20:16:13.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:13.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:13.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:13.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:13.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:13.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:13.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:13.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:14.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:13.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:14.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:13.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:14.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:13.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:14.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:13.939+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:14.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:13.709+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:13.710317+0000) 2024-02-19T20:16:14.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:13.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:14.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:13.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:14.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:13.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:14.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:13.935+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:14.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:14.237+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.238586+0000) 2024-02-19T20:16:14.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:14.448+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.449854+0000) 2024-02-19T20:16:14.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:14.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:14.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:14.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:14.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:14.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:14.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:14.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:15.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:14.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:15.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:14.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:15.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:14.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:15.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:14.945+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:15.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:14.711+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:14.712818+0000) 2024-02-19T20:16:15.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:14.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:15.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:14.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:15.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:14.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:15.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:14.891+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:15.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:15.241+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.242273+0000) 2024-02-19T20:16:15.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:15.451+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.452326+0000) 2024-02-19T20:16:15.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:15.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:15.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:15.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:15.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:15.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:15.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:15.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:16.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:15.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:16.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:15.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:16.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:15.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:16.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:15.929+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:16.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:15.713+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:15.714982+0000) 2024-02-19T20:16:16.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:15.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:16.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:15.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:16.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:15.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:16.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:15.890+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:16.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:16.241+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.242715+0000) 2024-02-19T20:16:16.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:16.451+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.452988+0000) 2024-02-19T20:16:16.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:16.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:16.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:16.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:16.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:16.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:16.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:16.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:16.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:16.714+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:16.715646+0000) 2024-02-19T20:16:16.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:16.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:16.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:16.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:16.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:16.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:16.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:16.864+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:17.243 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:17.008+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58393 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:17.243 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:17.006+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.007431+0000) 2024-02-19T20:16:17.244 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:17.005+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.006997+0000) 2024-02-19T20:16:17.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:16.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:17.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:16.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:17.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:16.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:17.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:16.975+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:17.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:17.005+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.006349+0000) 2024-02-19T20:16:17.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:17.241+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.242911+0000) 2024-02-19T20:16:17.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:17.452+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.453187+0000) 2024-02-19T20:16:17.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:17.714+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:17.715834+0000) 2024-02-19T20:16:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:17.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:17.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:17.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:17.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:17.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:18.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:18.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:18.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:18.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:18.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:18.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:18.161 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:18.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:18.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:17.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:18.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:17.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:18.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:17.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:18.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:17.882+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:18.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:18.244+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.245829+0000) 2024-02-19T20:16:18.709 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:18.454+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.455200+0000) 2024-02-19T20:16:18.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:18.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:18.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:18.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:18.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:18.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:18.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:18.605+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:19.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:18.717+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:18.718328+0000) 2024-02-19T20:16:19.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:18.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:19.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:18.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:19.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:18.910+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:19.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:18.911+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:19.247+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.248351+0000) 2024-02-19T20:16:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:19.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:19.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:19.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:19.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:19.709 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:19.456+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.457360+0000) 2024-02-19T20:16:19.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:19.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:19.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:19.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:19.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:19.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:19.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:19.627+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:20.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:19.719+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:19.720950+0000) 2024-02-19T20:16:20.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:19.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:20.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:19.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:20.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:19.919+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:20.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:19.919+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:20.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:20.250+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.251054+0000) 2024-02-19T20:16:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:20.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.458+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.459965+0000) 2024-02-19T20:16:20.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:20.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:20.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:20.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:20.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:20.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:20.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:20.641+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:20.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:20.723+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:20.724672+0000) 2024-02-19T20:16:20.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:20.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:20.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:20.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:20.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:20.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:20.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:20.883+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:21.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:21.251 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:21.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:21.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:20.984+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:21.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:21.251+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.251485+0000) 2024-02-19T20:16:21.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:21.459+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.460703+0000) 2024-02-19T20:16:21.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:21.725+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:21.725501+0000) 2024-02-19T20:16:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:21.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:21.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:21.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:21.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:21.623+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:22.160 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:22.015+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58432 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:22.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:22.013+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.014024+0000) 2024-02-19T20:16:22.161 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:21.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:22.161 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:21.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:22.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:21.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:22.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:21.945+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:22.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:22.012+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.013569+0000) 2024-02-19T20:16:22.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:21.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:22.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:21.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:22.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:21.885+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:22.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:21.885+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:22.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:22.012+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.013946+0000) 2024-02-19T20:16:22.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:22.250+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.251706+0000) 2024-02-19T20:16:22.725 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:22.459+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.460929+0000) 2024-02-19T20:16:22.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:22.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:22.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:22.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:22.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:22.667+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:22.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:22.668+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:23.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:22.724+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:22.725737+0000) 2024-02-19T20:16:23.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:22.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:23.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:22.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:23.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:22.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:23.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:22.843+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:23.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:22.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:23.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:22.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:23.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:22.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:23.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:22.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:23.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:23.252+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.253742+0000) 2024-02-19T20:16:23.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:23.462+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.463067+0000) 2024-02-19T20:16:23.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:23.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:23.729 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:23.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:23.729 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:23.652+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:23.729 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:23.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:23.728+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:23.728339+0000) 2024-02-19T20:16:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:23.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:23.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:23.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:23.848+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:24.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:23.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:24.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:23.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:24.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:23.925+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:24.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:23.925+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:24.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:24.255+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.255899+0000) 2024-02-19T20:16:24.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:24.465+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.465194+0000) 2024-02-19T20:16:24.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:24.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:24.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:24.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:24.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:24.622+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:24.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:24.622+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:25.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:24.730+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:24.730533+0000) 2024-02-19T20:16:25.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:24.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:25.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:24.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:25.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:24.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:25.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:24.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:25.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:24.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:25.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:24.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:25.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:24.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:25.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:24.910+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:25.257+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.258490+0000) 2024-02-19T20:16:25.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:25.466+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.467308+0000) 2024-02-19T20:16:25.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:25.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:25.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:25.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:25.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:25.638+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:25.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:25.638+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:26.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:25.733+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:25.734210+0000) 2024-02-19T20:16:26.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:25.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:26.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:25.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:26.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:25.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:26.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:25.814+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:26.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:25.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:26.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:25.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:26.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:25.877+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:26.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:25.878+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:26.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:26.258+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.259125+0000) 2024-02-19T20:16:26.734 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:26.466+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.467807+0000) 2024-02-19T20:16:26.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:26.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:26.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:26.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:26.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:26.651+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:26.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:26.652+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:27.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:26.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:27.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:26.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:27.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:26.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:27.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:26.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:26.733+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:26.734743+0000) 2024-02-19T20:16:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:26.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:26.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:26.789+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:26.789+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:27.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:27.022+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58472 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:27.020+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.021347+0000) 2024-02-19T20:16:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:27.020+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.021571+0000) 2024-02-19T20:16:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:27.021+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.022196+0000) 2024-02-19T20:16:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:27.258+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.259319+0000) 2024-02-19T20:16:27.735 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:27.466+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.467993+0000) 2024-02-19T20:16:27.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:27.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:27.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:27.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:27.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:27.660+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:27.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:27.660+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:28.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:27.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:28.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:27.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:28.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:27.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:28.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:27.921+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:28.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:27.733+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:27.734946+0000) 2024-02-19T20:16:28.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:27.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:28.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:27.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:28.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:27.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:28.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:27.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:28.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:28.261+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.262927+0000) 2024-02-19T20:16:28.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:28.470+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.471550+0000) 2024-02-19T20:16:28.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:28.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:28.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:28.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:28.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:28.697+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:28.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:28.698+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:29.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:28.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:29.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:28.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:29.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:28.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:29.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:28.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:29.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:28.736+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:28.737144+0000) 2024-02-19T20:16:29.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:28.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:29.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:28.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:29.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:28.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:29.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:28.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:29.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:29.264+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.265081+0000) 2024-02-19T20:16:29.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:29.473+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.474992+0000) 2024-02-19T20:16:29.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:29.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:29.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:29.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:29.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:29.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:29.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:29.680+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:30.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:29.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:30.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:29.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:30.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:29.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:30.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:29.940+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:30.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:29.738+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:29.739620+0000) 2024-02-19T20:16:30.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:29.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:30.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:29.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:30.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:29.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:30.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:29.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:30.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:30.477+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.478267+0000) 2024-02-19T20:16:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:30.266+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.267129+0000) 2024-02-19T20:16:30.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:30.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:30.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:30.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:30.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:30.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:30.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:30.658+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:30.741+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:30.742213+0000) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:30.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:30.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:30.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:30.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:30.832+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:31.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:30.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:31.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:30.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:31.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:30.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:31.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:30.945+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:31.478+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.479017+0000) 2024-02-19T20:16:31.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:31.266+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.267897+0000) 2024-02-19T20:16:31.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:31.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:31.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:31.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:31.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:31.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:31.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:31.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:31.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:31.741+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:31.742663+0000) 2024-02-19T20:16:31.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:31.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:31.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:31.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:31.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:31.854+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:31.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:31.855+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:32.268 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:32.029+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58511 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:32.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:32.026+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.027764+0000) 2024-02-19T20:16:32.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:32.026+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.027373+0000) 2024-02-19T20:16:32.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:31.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:32.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:31.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:32.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:31.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:32.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:31.981+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:32.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:32.026+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.027912+0000) 2024-02-19T20:16:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:32.478+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.479189+0000) 2024-02-19T20:16:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:32.267+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.268094+0000) 2024-02-19T20:16:32.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:32.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:32.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:32.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:32.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:32.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:32.962 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:32.690+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:32.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:32.741+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:32.742862+0000) 2024-02-19T20:16:32.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:32.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:32.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:32.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:32.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:32.865+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:32.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:32.866+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:32.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:32.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:32.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:33.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:32.961+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:33.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:33.480+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.481371+0000) 2024-02-19T20:16:33.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:33.269+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.270276+0000) 2024-02-19T20:16:33.973 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:33.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:33.973 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:33.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:33.973 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:33.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:33.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:33.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:33.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:33.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:33.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:33.648+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:33.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:33.648+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:33.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:33.744+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:33.745384+0000) 2024-02-19T20:16:33.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:33.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:33.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:33.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:33.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:33.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:33.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:33.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:34.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:33.972+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:34.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:34.483+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.484968+0000) 2024-02-19T20:16:34.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:34.271+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.272539+0000) 2024-02-19T20:16:34.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:34.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:34.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:34.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:34.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:34.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:34.939 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:34.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:34.939 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:34.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:34.939 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:34.659+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:34.939 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:34.659+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:34.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:34.747+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:34.748082+0000) 2024-02-19T20:16:34.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:34.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:34.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:34.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:34.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:34.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:34.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:34.875+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:35.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:34.937+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:35.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:35.487+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.488655+0000) 2024-02-19T20:16:35.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:35.273+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.274639+0000) 2024-02-19T20:16:35.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:35.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:35.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:35.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:35.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:35.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:35.907 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:35.630+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:35.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:35.749+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:35.750158+0000) 2024-02-19T20:16:35.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:35.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:35.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:35.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:35.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:35.847+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:35.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:35.848+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:36.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:35.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:36.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:35.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:36.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:35.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:36.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:35.905+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:36.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:36.274+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.275432+0000) 2024-02-19T20:16:36.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:36.488+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.489128+0000) 2024-02-19T20:16:36.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:36.650+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:36.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:36.650+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:36.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:36.650+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:36.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:36.650+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:37.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:36.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:37.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:36.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:37.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:36.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:37.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:36.861+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:37.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:36.749+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:36.750661+0000) 2024-02-19T20:16:37.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:36.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:37.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:36.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:37.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:36.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:37.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:36.896+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:37.275 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:37.037+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58551 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:37.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:37.034+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.035031+0000) 2024-02-19T20:16:37.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:37.034+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.035061+0000) 2024-02-19T20:16:37.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:37.034+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.035205+0000) 2024-02-19T20:16:37.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:37.488+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.489335+0000) 2024-02-19T20:16:37.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:37.274+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.275584+0000) 2024-02-19T20:16:38.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:37.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:38.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:37.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:38.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:37.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:38.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:37.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:38.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:37.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:38.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:37.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:38.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:37.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:38.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:37.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:38.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:37.749+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:37.750908+0000) 2024-02-19T20:16:38.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:37.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:38.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:37.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:38.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:37.887+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:38.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:37.888+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:38.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:38.490+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.491344+0000) 2024-02-19T20:16:38.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:38.277+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.278237+0000) 2024-02-19T20:16:39.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:38.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:39.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:38.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:39.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:38.893+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:39.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:38.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:39.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:38.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:39.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:38.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:39.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:38.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:39.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:38.677+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:39.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:38.752+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:38.753481+0000) 2024-02-19T20:16:39.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:38.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:39.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:38.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:39.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:38.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:39.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:38.879+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:39.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:39.279+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.280288+0000) 2024-02-19T20:16:39.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:39.492+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.493987+0000) 2024-02-19T20:16:40.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:39.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:40.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:39.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:40.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:39.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:40.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:39.900+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:40.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:39.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:40.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:39.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:40.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:39.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:40.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:39.706+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:40.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:39.755+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:39.756119+0000) 2024-02-19T20:16:40.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:39.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:40.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:39.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:40.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:39.877+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:40.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:39.878+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:40.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:40.495+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.496611+0000) 2024-02-19T20:16:40.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:40.281+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.282752+0000) 2024-02-19T20:16:40.915 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:40.916 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:40.916 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:40.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:40.916 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:40.734+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:40.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:40.757+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:40.758717+0000) 2024-02-19T20:16:40.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:40.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:40.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:40.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:40.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:40.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:40.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:40.874+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:41.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:40.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:41.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:40.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:41.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:40.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:41.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:40.915+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:41.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:41.496+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.497112+0000) 2024-02-19T20:16:41.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:41.282+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.283358+0000) 2024-02-19T20:16:42.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:41.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:42.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:41.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:42.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:41.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:42.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:41.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:42.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:41.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:42.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:41.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:42.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:41.715+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:42.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:41.715+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:42.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:41.758+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:41.759354+0000) 2024-02-19T20:16:42.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:41.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:42.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:41.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:42.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:41.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:42.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:41.862+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:42.283 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:42.044+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58590 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:42.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:42.041+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.042907+0000) 2024-02-19T20:16:42.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:42.041+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.043010+0000) 2024-02-19T20:16:42.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:42.042+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.043334+0000) 2024-02-19T20:16:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:42.496+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.497300+0000) 2024-02-19T20:16:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:42.282+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.283537+0000) 2024-02-19T20:16:43.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:42.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:43.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:42.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:43.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:42.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:43.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:42.880+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:43.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:42.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:43.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:42.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:43.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:42.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:43.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:42.707+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:43.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:42.758+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:42.759517+0000) 2024-02-19T20:16:43.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:42.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:43.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:42.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:43.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:42.906+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:43.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:42.906+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:43.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:43.286+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.287065+0000) 2024-02-19T20:16:43.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:43.499+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.500328+0000) 2024-02-19T20:16:44.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:43.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:44.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:43.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:44.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:43.708+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:44.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:43.709+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:44.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:43.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:44.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:43.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:44.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:43.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:44.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:43.858+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:44.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:43.760+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:43.761519+0000) 2024-02-19T20:16:44.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:43.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:44.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:43.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:44.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:43.889+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:44.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:43.890+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:44.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:44.501+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.502350+0000) 2024-02-19T20:16:44.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:44.289+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.290582+0000) 2024-02-19T20:16:45.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:44.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:45.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:44.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:45.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:44.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:45.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:44.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:45.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:44.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:45.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:44.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:45.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:44.897+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:45.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:44.898+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:45.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:44.762+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:44.763516+0000) 2024-02-19T20:16:45.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:44.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:45.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:44.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:45.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:44.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:45.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:44.918+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:45.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:45.503+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.504385+0000) 2024-02-19T20:16:45.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:45.293+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.294012+0000) 2024-02-19T20:16:46.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:45.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:46.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:45.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:46.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:45.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:46.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:45.908+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:46.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:45.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:46.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:45.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:46.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:45.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:46.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:45.693+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:46.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:45.764+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:45.765608+0000) 2024-02-19T20:16:46.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:45.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:46.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:45.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:46.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:45.892+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:46.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:45.892+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:46.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:46.293+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.294798+0000) 2024-02-19T20:16:46.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:46.504+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.505084+0000) 2024-02-19T20:16:46.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:46.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:46.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:46.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:46.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:46.742+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:46.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:46.743+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:46.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:46.765+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:46.766118+0000) 2024-02-19T20:16:46.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:46.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:46.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:46.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:46.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:46.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:46.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:46.912+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:47.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:47.052+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58630 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:47.048+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.049793+0000) 2024-02-19T20:16:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:47.048+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.049978+0000) 2024-02-19T20:16:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:46.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:46.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:46.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:46.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:47.049+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.050222+0000) 2024-02-19T20:16:47.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:47.293+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.294992+0000) 2024-02-19T20:16:47.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:47.504+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.505266+0000) 2024-02-19T20:16:47.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:47.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:47.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:47.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:47.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:47.713+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:47.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:47.713+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:47.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:47.765+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:47.766282+0000) 2024-02-19T20:16:47.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:47.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:47.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:47.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:47.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:47.931+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:47.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:47.932+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:47.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:47.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:47.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:47.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:48.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:48.506+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.507274+0000) 2024-02-19T20:16:48.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:48.297+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.298433+0000) 2024-02-19T20:16:48.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:48.767+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:48.768238+0000) 2024-02-19T20:16:48.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:48.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:48.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:48.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:48.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:48.926+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:48.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:48.926+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:48.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:48.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:48.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:48.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:48.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:48.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:48.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:48.723+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:49.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:48.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:48.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:48.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:48.987+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:49.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:49.508+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.509894+0000) 2024-02-19T20:16:49.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:49.299+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.300496+0000) 2024-02-19T20:16:49.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:49.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:49.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:49.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:49.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:49.754+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:49.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:49.755+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:49.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:49.769+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:49.770251+0000) 2024-02-19T20:16:49.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:49.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:49.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:49.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:49.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:49.945+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:50.211 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:49.945+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:50.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:50.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:50.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:50.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:50.211 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:50.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:50.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:50.015+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:50.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:50.301+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.302536+0000) 2024-02-19T20:16:50.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:50.511+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.512066+0000) 2024-02-19T20:16:50.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:50.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:50.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:50.736+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:50.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:50.737+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:51.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:50.771+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:50.772338+0000) 2024-02-19T20:16:51.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:50.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:51.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:50.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:51.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:50.915+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:51.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:50.915+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:51.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:51.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:51.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:51.034+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:51.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:51.302+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.303065+0000) 2024-02-19T20:16:51.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:51.511+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.512613+0000) 2024-02-19T20:16:52.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:51.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:52.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:51.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:52.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:51.690+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:52.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:51.690+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:52.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:51.771+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:51.772979+0000) 2024-02-19T20:16:52.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:51.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:52.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:51.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:52.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:51.934+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:52.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:51.935+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:52.057+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.058211+0000) 2024-02-19T20:16:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:52.056+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.057885+0000) 2024-02-19T20:16:52.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:52.059+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58669 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:52.056+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.057393+0000) 2024-02-19T20:16:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:52.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:52.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:52.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:52.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:52.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:52.511+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.512830+0000) 2024-02-19T20:16:52.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:52.302+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.303231+0000) 2024-02-19T20:16:52.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:52.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:52.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:52.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:52.675 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:52.673+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:52.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:52.673+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:52.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:52.772+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:52.773145+0000) 2024-02-19T20:16:52.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:52.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:52.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:52.959+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:53.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:52.960+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:53.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:53.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:53.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:53.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:53.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:53.515+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.516421+0000) 2024-02-19T20:16:53.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:53.304+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.305223+0000) 2024-02-19T20:16:53.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:53.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:53.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:53.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:53.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:53.666+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:53.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:53.667+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:53.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:53.774+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:53.775733+0000) 2024-02-19T20:16:53.998 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:53.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:53.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:53.996+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:54.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:53.997+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:54.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:54.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:54.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:54.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:54.125+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:54.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:54.519+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.520064+0000) 2024-02-19T20:16:54.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:54.306+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.307288+0000) 2024-02-19T20:16:54.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:54.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:54.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:54.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:54.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:54.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:55.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:54.670+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:55.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:54.777+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:54.778277+0000) 2024-02-19T20:16:55.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:55.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:55.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.004+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:55.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.005+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:55.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:55.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:55.159+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:55.160+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:55.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:55.521+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.522261+0000) 2024-02-19T20:16:55.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:55.308+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.309276+0000) 2024-02-19T20:16:55.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.779+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:55.780752+0000) 2024-02-19T20:16:55.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:55.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:55.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:55.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:55.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:55.669+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:55.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:55.670+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:56.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.968+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:56.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.968+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:56.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.968+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:56.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:55.969+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:56.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:56.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:56.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:56.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:56.678 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:56.522+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.523078+0000) 2024-02-19T20:16:56.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:56.308+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.309790+0000) 2024-02-19T20:16:56.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:56.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:56.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:56.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:56.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:56.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:56.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:56.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:56.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:56.780+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:56.781273+0000) 2024-02-19T20:16:56.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:56.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:56.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:56.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:56.944 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:56.942+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:57.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:16:57.067+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58709 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:16:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:57.064+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.065253+0000) 2024-02-19T20:16:57.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:56.943+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:57.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:57.063+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.065004+0000) 2024-02-19T20:16:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:57.063+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.064757+0000) 2024-02-19T20:16:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:57.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:57.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:57.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:57.112+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:57.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:57.522+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.523237+0000) 2024-02-19T20:16:57.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:57.309+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.310018+0000) 2024-02-19T20:16:57.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:57.780+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:57.781477+0000) 2024-02-19T20:16:57.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:57.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:57.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:57.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:57.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:57.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:57.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:57.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:57.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:57.716+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:57.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:57.716+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:57.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:58.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:57.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:58.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:58.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:58.073+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:58.074+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:58.678 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:58.525+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.526634+0000) 2024-02-19T20:16:58.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:58.311+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.312145+0000) 2024-02-19T20:16:58.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:58.783+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:58.784121+0000) 2024-02-19T20:16:58.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:58.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:58.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:58.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:58.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:58.975+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:58.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:58.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:58.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:58.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:58.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:58.677+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:58.978 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:58.678+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:59.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:58.976+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:16:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:59.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:59.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:59.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:16:59.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:59.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:16:59.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:59.313+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.314590+0000) 2024-02-19T20:16:59.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:16:59.529+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.530141+0000) 2024-02-19T20:16:59.988 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:59.785+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:16:59.786579+0000) 2024-02-19T20:16:59.989 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:59.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:59.989 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:59.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:59.989 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:59.987+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:16:59.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:59.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:59.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:59.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:59.990 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:59.692+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:16:59.990 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:16:59.693+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:00.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:16:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:16:59.988+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:00.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:00.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:00.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:00.144+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:00.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:00.317+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.318285+0000) 2024-02-19T20:17:00.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:00.531+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.532388+0000) 2024-02-19T20:17:00.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:00.787+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:00.788694+0000) 2024-02-19T20:17:00.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:00.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:00.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:00.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:00.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:00.990+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:00.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:00.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:00.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:00.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:00.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:00.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:00.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:00.682+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:00.991+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:01.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:01.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:01.111+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:01.111+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:01.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:01.532+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.533065+0000) 2024-02-19T20:17:01.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:01.317+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.318825+0000) 2024-02-19T20:17:01.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:01.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:01.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:01.788+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:01.789349+0000) 2024-02-19T20:17:01.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:01.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:01.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:01.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:01.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:01.657+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:01.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:01.657+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:02.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:02.071+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.072964+0000) 2024-02-19T20:17:02.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:02.074+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58748 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:02.070+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.071984+0000) 2024-02-19T20:17:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:02.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:02.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:02.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:02.093+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:01.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:01.949+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:02.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:01.950+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:02.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:02.070+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.071953+0000) 2024-02-19T20:17:02.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:02.532+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.533235+0000) 2024-02-19T20:17:02.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:02.318+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.319051+0000) 2024-02-19T20:17:02.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:02.788+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:02.789502+0000) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:02.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:02.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:02.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:02.954 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:02.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:02.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:02.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:02.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:02.647+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:02.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:02.647+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:03.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:02.953+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:03.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:03.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:03.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:03.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:03.672 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:03.535+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.536234+0000) 2024-02-19T20:17:03.672 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:03.320+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.321442+0000) 2024-02-19T20:17:03.672 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:03.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:03.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:03.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:03.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:03.670+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:03.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:03.790+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:03.791656+0000) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:03.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:03.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:03.953 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:03.951+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:04.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:03.952+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:04.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:04.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:04.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:04.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:04.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:04.537+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.538403+0000) 2024-02-19T20:17:04.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:04.322+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.323532+0000) 2024-02-19T20:17:04.971 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:04.792+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:04.793668+0000) 2024-02-19T20:17:04.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:04.703+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:04.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:04.703+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:04.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:04.703+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:04.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:04.704+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:05.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:04.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:04.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:04.970+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:04.971+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:05.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:05.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:05.151+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:05.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:05.152+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:05.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:05.324+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.325579+0000) 2024-02-19T20:17:05.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:05.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:05.663 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:05.539+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.540427+0000) 2024-02-19T20:17:06.016 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:05.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:06.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:05.662+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:06.017 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:05.662+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:06.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:05.795+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:05.796212+0000) 2024-02-19T20:17:06.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.015+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:06.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.015+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:06.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.015+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.016+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:06.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:06.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:06.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:06.174+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:06.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:06.539+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.540975+0000) 2024-02-19T20:17:06.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:06.325+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.326294+0000) 2024-02-19T20:17:06.979 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.795+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:06.796738+0000) 2024-02-19T20:17:06.979 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:06.980 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:06.980 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:06.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:06.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:06.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:06.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:06.681+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:07.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:07.080+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58788 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:07.078+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.079758+0000) 2024-02-19T20:17:07.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:06.979+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:07.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:07.078+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.079128+0000) 2024-02-19T20:17:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:07.078+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.079509+0000) 2024-02-19T20:17:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:07.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:07.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:07.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:07.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:07.157+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:07.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:07.540+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.541160+0000) 2024-02-19T20:17:07.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:07.325+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.326506+0000) 2024-02-19T20:17:08.020 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:07.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:08.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:07.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:08.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:07.725+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:08.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:07.725+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:08.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:07.795+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:07.796955+0000) 2024-02-19T20:17:08.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:07.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:08.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:07.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:08.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:07.937+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:08.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:07.937+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:08.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:08.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:08.128+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:08.129+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:08.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:08.542+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.543158+0000) 2024-02-19T20:17:08.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:08.327+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.328511+0000) 2024-02-19T20:17:09.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:08.797+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:08.798896+0000) 2024-02-19T20:17:09.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:08.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:09.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:08.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:09.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:08.909+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:09.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:08.910+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:09.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:08.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:09.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:08.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:09.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:08.773+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:09.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:08.773+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:09.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:09.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:09.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:09.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:09.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:09.544+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.545182+0000) 2024-02-19T20:17:09.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:09.329+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.330587+0000) 2024-02-19T20:17:10.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:09.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:10.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:09.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:10.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:09.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:10.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:09.765+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:10.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:09.799+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:09.800877+0000) 2024-02-19T20:17:10.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:09.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:10.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:09.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:10.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:09.881+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:10.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:09.881+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:10.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:10.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:10.157+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:10.159+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:10.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:10.548+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.548781+0000) 2024-02-19T20:17:10.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:10.332+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.332655+0000) 2024-02-19T20:17:11.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:10.802+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:10.802936+0000) 2024-02-19T20:17:11.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:10.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:11.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:10.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:11.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:10.917+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:11.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:10.918+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:11.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:10.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:11.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:10.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:11.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:10.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:11.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:10.799+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:11.185+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:11.185+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:11.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:11.186+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:11.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:11.548+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.549640+0000) 2024-02-19T20:17:11.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:11.332+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.333223+0000) 2024-02-19T20:17:11.919 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:11.802+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:11.803556+0000) 2024-02-19T20:17:11.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:11.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:11.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:11.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:11.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:11.918+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:11.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:11.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:11.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:11.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:11.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:11.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:11.921 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:11.794+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:12.208 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:12.088+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58828 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:12.209 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:12.086+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.087105+0000) 2024-02-19T20:17:12.209 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:11.919+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:12.209 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:12.084+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.085908+0000) 2024-02-19T20:17:12.210 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:12.084+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.085856+0000) 2024-02-19T20:17:12.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:12.332+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.333417+0000) 2024-02-19T20:17:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:12.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:12.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:12.227+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:12.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:12.228+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:12.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:12.548+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.549860+0000) 2024-02-19T20:17:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:12.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:12.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:12.751+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:12.752+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:13.165 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:12.802+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:12.803768+0000) 2024-02-19T20:17:13.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:12.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:13.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:12.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:13.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:12.891+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:13.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:12.892+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:13.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:13.334+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.335427+0000) 2024-02-19T20:17:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:13.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:13.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:13.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:13.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:13.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:13.807 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:13.551+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.552008+0000) 2024-02-19T20:17:13.807 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:13.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:13.807 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:13.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:13.807 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:13.723+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:13.808 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:13.724+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:14.165 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:13.806+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:13.807085+0000) 2024-02-19T20:17:14.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:13.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:14.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:13.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:14.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:13.879+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:14.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:13.880+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:14.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:14.336+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.337831+0000) 2024-02-19T20:17:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:14.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:14.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:14.276+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:14.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:14.277+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:14.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:14.553+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.554069+0000) 2024-02-19T20:17:14.810 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:14.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:14.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:14.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:14.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:14.729+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:14.811 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:14.729+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:15.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:14.809+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:14.810374+0000) 2024-02-19T20:17:15.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:14.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:15.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:14.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:15.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:14.882+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:15.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:14.883+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:15.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:15.338+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.339884+0000) 2024-02-19T20:17:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:15.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:15.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:15.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:15.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:15.238+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:15.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:15.555+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.556803+0000) 2024-02-19T20:17:15.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:15.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:15.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:15.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:15.813 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:15.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:15.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:15.683+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:16.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:15.812+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:15.813165+0000) 2024-02-19T20:17:16.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:15.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:16.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:15.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:16.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:15.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:16.166 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:15.856+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:16.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:16.339+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.340798+0000) 2024-02-19T20:17:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:16.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:16.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:16.254+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:16.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:16.255+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:16.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:16.556+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.557505+0000) 2024-02-19T20:17:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:16.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:16.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:16.695+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:16.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:16.695+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:17.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:16.812+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:16.813838+0000) 2024-02-19T20:17:17.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:16.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:17.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:16.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:17.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:16.842+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:17.091 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:16.843+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:17.341 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:17.095+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58868 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:17.341 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:17.093+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.094556+0000) 2024-02-19T20:17:17.341 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:17.092+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.093761+0000) 2024-02-19T20:17:17.342 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:17.093+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.094535+0000) 2024-02-19T20:17:17.342 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:17.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:17.342 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:17.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:17.342 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:17.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:17.342 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:17.298+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:17.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:17.339+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.341001+0000) 2024-02-19T20:17:17.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:17.556+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.557669+0000) 2024-02-19T20:17:18.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:17.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:18.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:17.703+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:18.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:17.703+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:18.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:17.703+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:18.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:17.813+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:17.814083+0000) 2024-02-19T20:17:18.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:17.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:18.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:17.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:18.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:17.860+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:18.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:17.861+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:18.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:18.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:18.656 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:18.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:18.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:18.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:18.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:18.343+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:18.657 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:18.558+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.559753+0000) 2024-02-19T20:17:18.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:18.342+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.343587+0000) 2024-02-19T20:17:18.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:18.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:18.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:18.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:18.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:18.655+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:19.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:18.655+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:19.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:18.815+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:18.816096+0000) 2024-02-19T20:17:19.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:18.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:19.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:18.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:19.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:18.817+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:19.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:18.817+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:19.608 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:19.344+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.345899+0000) 2024-02-19T20:17:19.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:19.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:19.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:19.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:19.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:19.378+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:19.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:19.379+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:19.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:19.562+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.563338+0000) 2024-02-19T20:17:20.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:19.817+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:19.818387+0000) 2024-02-19T20:17:20.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:19.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:20.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:19.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:20.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:19.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:20.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:19.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:20.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:19.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:20.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:19.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:20.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:19.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:20.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:19.607+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:20.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:20.346+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.347960+0000) 2024-02-19T20:17:20.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:20.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:20.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:20.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:20.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:20.623+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:20.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:20.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:20.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:20.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:20.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:20.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:20.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:20.382+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:20.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:20.565+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.566945+0000) 2024-02-19T20:17:21.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:20.624+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:21.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:20.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:21.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:20.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:21.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:20.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:21.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:20.789+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:21.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:20.819+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:20.820485+0000) 2024-02-19T20:17:21.629 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:21.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:21.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:21.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:21.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:21.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:21.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:21.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:21.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:21.566+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.567548+0000) 2024-02-19T20:17:21.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:21.347+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.348888+0000) 2024-02-19T20:17:21.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:21.628+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:21.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:21.628+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:21.631 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:21.628+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:22.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:21.629+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:22.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:21.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:22.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:21.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:22.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:21.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:22.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:21.788+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:22.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:21.820+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:21.821069+0000) 2024-02-19T20:17:22.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:22.099+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.100222+0000) 2024-02-19T20:17:22.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:22.102+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58907 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:22.099+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.100427+0000) 2024-02-19T20:17:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:22.099+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.100606+0000) 2024-02-19T20:17:22.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:22.348+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.349045+0000) 2024-02-19T20:17:22.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:22.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:22.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:22.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:22.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:22.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:22.607 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:22.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:22.607 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:22.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:22.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:22.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:22.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:22.348+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:22.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:22.566+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.567815+0000) 2024-02-19T20:17:23.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:22.606+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:23.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:22.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:23.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:22.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:23.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:22.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:23.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:22.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:23.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:22.820+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:22.821227+0000) 2024-02-19T20:17:23.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:23.350+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.351045+0000) 2024-02-19T20:17:23.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:23.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:23.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:23.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:23.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:23.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:23.599 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:23.323+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:23.599 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:23.324+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:23.599 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:23.569+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.570013+0000) 2024-02-19T20:17:24.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:23.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:24.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:23.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:24.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:23.597+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:24.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:23.822+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:23.823698+0000) 2024-02-19T20:17:24.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:23.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:24.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:23.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:24.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:23.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:24.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:23.838+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:24.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:24.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:24.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:24.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:24.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:24.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:24.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:24.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:24.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:24.572+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.573624+0000) 2024-02-19T20:17:24.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:24.352+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.353470+0000) 2024-02-19T20:17:24.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:24.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:24.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:24.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:24.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:24.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:24.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:24.562+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:25.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:24.825+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:24.826260+0000) 2024-02-19T20:17:25.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:24.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:25.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:24.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:25.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:24.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:25.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:24.830+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:25.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:25.354+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.355792+0000) 2024-02-19T20:17:25.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:25.544+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:25.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:25.544+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:25.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:25.544+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:25.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:25.545+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:25.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:25.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:25.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:25.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:25.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:25.371+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:25.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:25.372+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:25.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:25.576+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.577214+0000) 2024-02-19T20:17:26.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:25.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:26.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:25.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:26.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:25.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:26.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:25.827+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:25.828478+0000) 2024-02-19T20:17:26.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:25.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:26.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:26.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:26.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:26.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:26.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:26.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:26.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:26.406+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:26.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:26.576+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.577852+0000) 2024-02-19T20:17:26.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:26.355+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.356803+0000) 2024-02-19T20:17:26.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:26.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:26.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:26.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:26.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:26.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:26.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:26.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:27.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:26.828+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:26.829277+0000) 2024-02-19T20:17:27.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:26.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:27.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:26.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:27.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:26.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:27.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:26.858+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:27.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:27.107+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.108920+0000) 2024-02-19T20:17:27.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:27.109+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58947 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:27.106+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.107606+0000) 2024-02-19T20:17:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:27.106+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.107775+0000) 2024-02-19T20:17:27.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:27.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:27.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:27.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:27.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:27.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:27.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:27.380+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:27.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:27.577+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.578112+0000) 2024-02-19T20:17:27.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:27.356+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.357041+0000) 2024-02-19T20:17:27.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:27.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:27.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:27.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:27.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:27.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:27.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:27.540+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:28.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:27.828+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:27.829460+0000) 2024-02-19T20:17:28.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:27.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:28.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:27.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:28.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:27.844+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:28.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:27.845+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:28.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:28.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:28.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:28.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:28.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:28.341+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:28.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:28.342+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:28.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:28.579+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.580877+0000) 2024-02-19T20:17:28.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:28.359+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.360435+0000) 2024-02-19T20:17:28.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:28.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:28.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:28.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:28.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:28.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:28.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:28.547+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:29.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:28.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:29.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:28.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:29.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:28.810+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:29.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:28.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:29.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:28.830+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:28.831969+0000) 2024-02-19T20:17:29.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:29.362+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.363143+0000) 2024-02-19T20:17:29.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:29.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:29.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:29.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:29.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:29.303+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:29.557 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:29.303+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:29.834 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:29.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:29.835 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:29.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:29.835 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:29.788+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:29.835 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:29.788+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:29.835 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:29.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:29.835 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:29.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:29.835 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:29.579+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:29.836 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:29.579+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:29.836 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:29.582+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.583374+0000) 2024-02-19T20:17:30.167 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:29.833+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:29.834546+0000) 2024-02-19T20:17:30.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:30.364+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.365605+0000) 2024-02-19T20:17:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:30.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:30.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:30.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:30.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:30.273+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:30.836 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:30.585+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.586080+0000) 2024-02-19T20:17:30.837 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:30.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:30.837 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:30.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:30.837 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:30.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:30.837 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:30.750+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:30.837 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:30.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:30.837 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:30.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:30.838 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:30.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:30.838 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:30.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:31.167 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:30.835+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:30.836654+0000) 2024-02-19T20:17:31.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:31.365+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.366464+0000) 2024-02-19T20:17:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:31.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:31.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:31.318+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:31.319+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:31.837 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:31.585+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.586978+0000) 2024-02-19T20:17:31.838 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:31.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:31.838 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:31.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:31.838 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:31.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:31.838 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:31.585+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:31.838 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:31.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:31.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:31.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:31.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:31.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:31.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:31.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:32.111 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:31.836+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:31.837345+0000) 2024-02-19T20:17:32.363 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:32.114+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.115710+0000) 2024-02-19T20:17:32.364 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:32.113+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.114571+0000) 2024-02-19T20:17:32.364 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:32.113+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.114941+0000) 2024-02-19T20:17:32.364 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:32.116+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 58986 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:32.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:32.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:32.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:32.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:32.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:32.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:32.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:32.363+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:32.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:32.586+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.587143+0000) 2024-02-19T20:17:32.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:32.365+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.366663+0000) 2024-02-19T20:17:32.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:32.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:32.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:32.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:32.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:32.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:32.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:32.597+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:33.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:32.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:33.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:32.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:33.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:32.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:33.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:32.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:33.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:32.836+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:32.837581+0000) 2024-02-19T20:17:33.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:33.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:33.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:33.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:33.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:33.349+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:33.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:33.350+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:33.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:33.588+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.589182+0000) 2024-02-19T20:17:33.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:33.369+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.370278+0000) 2024-02-19T20:17:33.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:33.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:33.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:33.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:33.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:33.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:33.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:33.570+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:34.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:33.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:34.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:33.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:34.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:33.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:34.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:33.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:34.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:33.840+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:33.841097+0000) 2024-02-19T20:17:34.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:34.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:34.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:34.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:34.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:34.340+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:34.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:34.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:34.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:34.590+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.591241+0000) 2024-02-19T20:17:34.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:34.371+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.372334+0000) 2024-02-19T20:17:34.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:34.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:34.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:34.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:34.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:34.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:34.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:34.542+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:35.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:34.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:35.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:34.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:35.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:34.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:35.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:34.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:35.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:34.842+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:34.843236+0000) 2024-02-19T20:17:35.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:35.373+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.374350+0000) 2024-02-19T20:17:35.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:35.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:35.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:35.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:35.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:35.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:35.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:35.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:35.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:35.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:35.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:35.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:35.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:35.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:35.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:35.369+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:35.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:35.592+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.593285+0000) 2024-02-19T20:17:36.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:35.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:36.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:35.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:36.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:35.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:36.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:35.743+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:36.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:35.844+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:35.845258+0000) 2024-02-19T20:17:36.594 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:36.373+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.374986+0000) 2024-02-19T20:17:36.594 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:36.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:36.594 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:36.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:36.594 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:36.518+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:36.594 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:36.519+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:36.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:36.320+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:36.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:36.320+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:36.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:36.320+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:36.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:36.321+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:36.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:36.592+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.593899+0000) 2024-02-19T20:17:36.846 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:36.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:36.846 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:36.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:36.846 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:36.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:36.846 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:36.756+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:37.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:36.844+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:36.845861+0000) 2024-02-19T20:17:37.375 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:37.124+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59026 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:37.376 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:37.120+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.121846+0000) 2024-02-19T20:17:37.376 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:37.120+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.121703+0000) 2024-02-19T20:17:37.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:37.121+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.122419+0000) 2024-02-19T20:17:37.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:37.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:37.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:37.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:37.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:37.272+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:37.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:37.272+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:37.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:37.593+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.594103+0000) 2024-02-19T20:17:37.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:37.374+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.375188+0000) 2024-02-19T20:17:37.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:37.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:37.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:37.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:37.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:37.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:37.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:37.532+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:38.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:37.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:38.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:37.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:38.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:37.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:08.081491+0000 (oldest deadline 2024-02-19T20:07:28.081491+0000) 2024-02-19T20:17:38.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:37.802+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:38.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:37.845+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:37.846089+0000) 2024-02-19T20:17:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:38.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:38.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:38.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:38.246+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:38.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:38.595+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.596734+0000) 2024-02-19T20:17:38.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:38.376+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.377625+0000) 2024-02-19T20:17:38.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:38.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:38.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:38.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:38.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:38.569+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:38.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:38.569+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:39.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:38.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:39.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:38.848+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:38.849687+0000) 2024-02-19T20:17:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:39.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:39.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:39.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:39.244+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:39.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:39.597+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.598882+0000) 2024-02-19T20:17:39.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:39.379+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.380067+0000) 2024-02-19T20:17:39.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:39.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:39.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:39.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:39.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:39.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:39.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:39.577+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:40.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:39.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:40.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:39.850+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:39.851745+0000) 2024-02-19T20:17:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:40.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:40.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:40.266+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:40.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:40.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:40.601+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.602443+0000) 2024-02-19T20:17:40.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:40.381+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.382736+0000) 2024-02-19T20:17:40.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:40.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:40.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:40.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:40.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:40.584+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:40.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:40.584+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:41.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:40.716+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:41.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:40.852+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:40.853858+0000) 2024-02-19T20:17:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:41.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:41.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:41.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:41.236+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:41.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:41.602+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.603265+0000) 2024-02-19T20:17:41.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:41.382+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.383427+0000) 2024-02-19T20:17:41.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:41.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:41.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:41.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:41.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:41.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:41.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:41.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:42.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:41.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:42.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:41.853+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:41.854704+0000) 2024-02-19T20:17:42.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:42.128+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.129244+0000) 2024-02-19T20:17:42.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:42.132+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59065 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:42.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:42.127+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.128494+0000) 2024-02-19T20:17:42.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:42.129+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.130094+0000) 2024-02-19T20:17:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:42.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:42.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:42.235+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:42.236+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:42.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:42.382+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.383629+0000) 2024-02-19T20:17:42.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:42.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:42.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:42.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:42.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:42.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:42.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:42.506+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:42.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:42.717+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:42.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:42.602+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.603455+0000) 2024-02-19T20:17:43.168 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:42.853+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:42.854926+0000) 2024-02-19T20:17:43.458 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:43.385+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.386261+0000) 2024-02-19T20:17:43.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:43.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:43.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:43.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:43.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:43.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:43.459 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:43.257+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:43.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:43.605+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.606848+0000) 2024-02-19T20:17:43.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:43.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:43.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:43.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:43.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:43.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:43.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:43.458+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:44.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:43.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:44.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:43.856+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:43.857042+0000) 2024-02-19T20:17:44.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:44.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:44.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:44.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:44.217+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:44.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:44.609+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.610475+0000) 2024-02-19T20:17:44.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:44.387+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.388282+0000) 2024-02-19T20:17:44.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:44.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:44.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:44.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:44.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:44.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:44.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:44.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:45.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:44.788+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:45.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:44.858+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:44.859064+0000) 2024-02-19T20:17:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:45.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:45.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:45.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:45.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:45.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:45.612+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.613956+0000) 2024-02-19T20:17:45.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:45.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:45.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:45.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:45.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:45.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:45.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:45.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:45.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:45.389+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.390302+0000) 2024-02-19T20:17:46.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:45.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:46.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:45.861+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:45.862521+0000) 2024-02-19T20:17:46.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:46.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:46.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:46.244+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:46.245+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:46.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:46.346+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:46.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:46.346+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:46.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:46.346+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:46.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:46.347+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:46.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:46.389+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.390940+0000) 2024-02-19T20:17:47.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:46.614+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.615013+0000) 2024-02-19T20:17:47.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:46.806+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:47.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:46.862+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:46.863522+0000) 2024-02-19T20:17:47.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:47.139+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59102 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:47.135+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.136876+0000) 2024-02-19T20:17:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:47.136+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.137519+0000) 2024-02-19T20:17:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:47.136+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.137533+0000) 2024-02-19T20:17:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:47.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:47.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:47.255+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:47.256+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:47.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:47.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:47.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:47.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:47.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:47.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:47.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:47.330+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:47.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:47.390+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.391156+0000) 2024-02-19T20:17:48.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:47.614+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.615221+0000) 2024-02-19T20:17:48.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:47.787+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:48.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:47.862+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:47.863676+0000) 2024-02-19T20:17:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:48.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:48.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:48.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:48.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:48.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:48.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:48.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:48.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:48.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:48.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:48.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:48.327+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:48.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:48.392+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.393104+0000) 2024-02-19T20:17:49.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:48.616+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.617652+0000) 2024-02-19T20:17:49.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:48.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:49.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:48.864+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:48.865678+0000) 2024-02-19T20:17:49.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:49.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:49.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:49.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:49.237+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:49.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:49.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:49.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:49.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:49.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:49.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:49.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:49.339+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:49.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:49.394+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.395118+0000) 2024-02-19T20:17:50.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:49.619+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.620050+0000) 2024-02-19T20:17:50.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:49.819+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:50.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:49.867+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:49.868212+0000) 2024-02-19T20:17:50.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:50.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:50.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:50.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:50.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:50.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:50.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:50.217+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:50.572 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:50.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:50.573 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:50.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:50.573 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:50.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:50.573 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:50.297+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:50.573 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:50.396+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.397756+0000) 2024-02-19T20:17:50.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:50.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:50.870 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:50.621+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.622532+0000) 2024-02-19T20:17:51.169 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:50.869+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:50.870151+0000) 2024-02-19T20:17:51.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:51.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:51.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:51.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:51.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:51.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:51.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:51.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:51.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:51.397+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.398790+0000) 2024-02-19T20:17:51.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:51.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:51.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:51.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:51.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:51.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:51.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:51.171+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:51.861 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:51.622+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.623187+0000) 2024-02-19T20:17:52.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:51.860+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:52.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:51.869+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:51.870886+0000) 2024-02-19T20:17:52.399 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:52.146+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59141 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:52.399 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:52.143+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.144117+0000) 2024-02-19T20:17:52.399 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:52.143+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.144188+0000) 2024-02-19T20:17:52.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:52.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:52.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:52.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:52.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:52.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:18.770483+0000 (oldest deadline 2024-02-19T20:07:38.770483+0000) 2024-02-19T20:17:52.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:52.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:52.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:52.143+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.144839+0000) 2024-02-19T20:17:52.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:52.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:52.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:52.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:52.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:52.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:52.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:52.198+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:52.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:52.397+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.398976+0000) 2024-02-19T20:17:52.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:52.622+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.623416+0000) 2024-02-19T20:17:53.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:52.870+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:52.871061+0000) 2024-02-19T20:17:53.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:52.891+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:53.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:53.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:53.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:53.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:53.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:53.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:53.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:53.576 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:53.400+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.401544+0000) 2024-02-19T20:17:53.873 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:53.624+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.625879+0000) 2024-02-19T20:17:54.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:53.872+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:53.873453+0000) 2024-02-19T20:17:54.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:53.898+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:54.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:54.309+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:54.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:54.403+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.403812+0000) 2024-02-19T20:17:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:54.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:54.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:54.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:54.216+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:54.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:54.216+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:54.872 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:54.628+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.629466+0000) 2024-02-19T20:17:55.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:54.871+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:55.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:54.874+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:54.875985+0000) 2024-02-19T20:17:55.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:55.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:55.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:55.406+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.407492+0000) 2024-02-19T20:17:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:55.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:55.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:55.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:55.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:55.223+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:55.879 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:55.631+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.632017+0000) 2024-02-19T20:17:56.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:55.879+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:55.879963+0000) 2024-02-19T20:17:56.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:55.918+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:56.397+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:56.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:56.407+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.408252+0000) 2024-02-19T20:17:56.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:56.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:56.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:56.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:56.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:56.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:56.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:56.260+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:56.880 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:56.633+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.633095+0000) 2024-02-19T20:17:57.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:56.880+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:56.880612+0000) 2024-02-19T20:17:57.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:56.924+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:57.408 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:17:57.154+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59181 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:17:57.409 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:57.152+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.152401+0000) 2024-02-19T20:17:57.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:57.151+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.151238+0000) 2024-02-19T20:17:57.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:57.152+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.152990+0000) 2024-02-19T20:17:57.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:57.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:57.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:57.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:57.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:57.233+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:07:22.042863+0000 (oldest deadline 2024-02-19T20:07:42.042863+0000) 2024-02-19T20:17:57.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:57.234+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:57.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:57.632+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.633261+0000) 2024-02-19T20:17:57.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:57.407+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.408433+0000) 2024-02-19T20:17:57.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:57.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:58.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:57.879+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:57.880804+0000) 2024-02-19T20:17:58.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:57.933+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:58.251+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:58.698 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:58.634+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.635271+0000) 2024-02-19T20:17:58.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:58.409+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.410450+0000) 2024-02-19T20:17:58.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:58.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:59.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:58.882+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:58.883306+0000) 2024-02-19T20:17:59.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:58.933+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:17:59.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:59.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:17:59.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:17:59.636+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.637376+0000) 2024-02-19T20:17:59.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:59.413+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.414118+0000) 2024-02-19T20:17:59.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:17:59.424+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:00.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:59.884+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:17:59.885737+0000) 2024-02-19T20:18:00.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:17:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:17:59.893+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:00.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:00.197+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:00.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:00.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:00.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:00.416+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.417784+0000) 2024-02-19T20:18:01.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:00.638+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.639447+0000) 2024-02-19T20:18:01.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:00.864+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:01.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:00.887+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:00.888208+0000) 2024-02-19T20:18:01.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:01.169+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:01.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:01.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:01.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:01.417+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.418430+0000) 2024-02-19T20:18:02.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:01.639+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.640468+0000) 2024-02-19T20:18:02.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:01.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:02.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:01.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:02.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:01.846+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:02.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:01.847+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:02.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:01.887+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:01.888844+0000) 2024-02-19T20:18:02.281 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:02.158+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.159327+0000) 2024-02-19T20:18:02.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:02.158+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.159430+0000) 2024-02-19T20:18:02.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:02.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:02.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:02.158+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.159324+0000) 2024-02-19T20:18:02.282 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:02.160+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59220 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:02.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:02.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:02.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:02.417+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.418615+0000) 2024-02-19T20:18:02.851 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:02.639+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.640693+0000) 2024-02-19T20:18:03.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:02.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:03.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:02.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:03.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:02.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:03.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:02.851+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:03.150 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:02.888+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:02.889058+0000) 2024-02-19T20:18:03.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:03.356+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:03.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:03.147+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:03.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:03.419+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.420804+0000) 2024-02-19T20:18:03.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:03.641+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.642881+0000) 2024-02-19T20:18:04.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:03.890+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:03.891076+0000) 2024-02-19T20:18:04.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:03.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:04.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:03.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:04.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:03.898+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:04.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:03.898+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:04.134+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:04.584 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:04.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:04.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:04.422+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.423059+0000) 2024-02-19T20:18:04.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:04.644+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.645063+0000) 2024-02-19T20:18:05.171 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:05.125+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:05.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:04.892+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:04.893094+0000) 2024-02-19T20:18:05.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:04.903+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:05.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:04.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:05.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:04.904+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:05.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:04.904+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:05.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:05.339+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:05.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:05.646+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.647212+0000) 2024-02-19T20:18:05.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:05.424+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.425700+0000) 2024-02-19T20:18:05.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:05.894+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:05.895227+0000) 2024-02-19T20:18:05.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:05.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:05.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:05.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:05.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:05.954+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:06.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:05.955+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:06.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:06.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:06.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:06.321+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:06.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:06.425+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.426701+0000) 2024-02-19T20:18:06.895 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:06.646+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.647952+0000) 2024-02-19T20:18:07.163 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:07.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:07.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:06.894+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:06.895837+0000) 2024-02-19T20:18:07.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:06.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:07.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:06.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:07.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:06.981+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:07.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:06.981+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:07.427 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:07.168+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59260 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:07.427 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:07.164+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.165982+0000) 2024-02-19T20:18:07.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:07.164+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.165982+0000) 2024-02-19T20:18:07.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:07.165+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.166354+0000) 2024-02-19T20:18:07.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:07.275+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:07.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:07.647+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.648180+0000) 2024-02-19T20:18:07.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:07.425+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.426941+0000) 2024-02-19T20:18:08.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:07.894+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:07.895989+0000) 2024-02-19T20:18:08.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:08.018+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:08.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:08.018+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:08.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:08.018+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:08.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:08.019+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:08.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:08.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:08.282+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:08.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:08.428+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.429442+0000) 2024-02-19T20:18:08.898 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:08.649+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.650188+0000) 2024-02-19T20:18:09.171 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:09.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:09.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:08.897+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:08.898602+0000) 2024-02-19T20:18:09.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:09.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:09.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:09.172 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.021+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:09.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:09.236+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:09.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:09.652+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.653778+0000) 2024-02-19T20:18:09.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:09.430+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.431914+0000) 2024-02-19T20:18:09.979 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.899+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:09.900958+0000) 2024-02-19T20:18:09.980 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:09.980 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:09.980 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.978+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:10.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:09.979+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:10.111+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:10.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:10.205+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:10.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:10.433+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.434375+0000) 2024-02-19T20:18:10.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:10.656+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.657374+0000) 2024-02-19T20:18:10.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:10.901+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:10.902977+0000) 2024-02-19T20:18:10.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:10.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:10.963 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:10.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:11.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:11.169+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:10.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:10.962+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:11.700 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:11.657+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.658206+0000) 2024-02-19T20:18:11.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:11.434+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.435453+0000) 2024-02-19T20:18:12.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:11.902+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:11.903663+0000) 2024-02-19T20:18:12.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:11.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:12.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:11.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:12.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:11.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:12.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:11.914+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:12.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:12.172+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.173785+0000) 2024-02-19T20:18:12.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:12.176+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59299 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:12.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:12.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:12.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:12.172+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.173597+0000) 2024-02-19T20:18:12.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:12.173+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.174723+0000) 2024-02-19T20:18:12.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:12.217+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:12.589 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:12.434+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.435634+0000) 2024-02-19T20:18:12.875 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:12.657+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.658415+0000) 2024-02-19T20:18:13.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:13.112+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:12.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:12.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:12.874+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:12.874+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:13.222 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:12.902+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:12.903852+0000) 2024-02-19T20:18:13.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:13.220+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:13.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:13.437+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.438085+0000) 2024-02-19T20:18:13.864 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:13.660+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.661084+0000) 2024-02-19T20:18:14.147 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:13.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:14.147 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:13.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:14.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:13.863+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:14.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:13.864+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:14.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:13.906+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:13.907270+0000) 2024-02-19T20:18:14.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:14.146+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:14.440 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:14.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:14.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:14.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:14.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:14.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:14.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:14.196+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:14.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:14.439+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.440113+0000) 2024-02-19T20:18:14.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:14.662+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.663132+0000) 2024-02-19T20:18:15.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:14.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:15.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:14.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:15.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:14.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:15.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:14.837+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:15.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:14.909+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:14.910749+0000) 2024-02-19T20:18:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:15.111+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:15.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:15.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:15.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:15.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:15.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:15.441+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.442287+0000) 2024-02-19T20:18:15.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:15.664+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.665545+0000) 2024-02-19T20:18:16.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:15.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:16.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:15.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:16.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:15.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:16.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:15.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:16.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:15.913+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:15.914251+0000) 2024-02-19T20:18:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:16.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:16.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:16.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:16.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:16.202+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:16.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:16.442+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.443261+0000) 2024-02-19T20:18:16.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:16.665+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.666229+0000) 2024-02-19T20:18:17.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:16.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:17.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:16.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:17.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:16.773+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:17.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:16.774+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:17.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:16.914+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:16.915324+0000) 2024-02-19T20:18:17.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:17.183+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59339 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:17.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:17.180+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.181939+0000) 2024-02-19T20:18:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:17.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:17.180+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.181313+0000) 2024-02-19T20:18:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:17.166+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:17.166+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:17.166+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:17.166+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:17.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:17.182+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.183221+0000) 2024-02-19T20:18:17.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:17.442+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.443442+0000) 2024-02-19T20:18:17.915 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:17.665+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.666419+0000) 2024-02-19T20:18:17.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:17.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:17.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:17.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:17.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:17.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:17.916 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:17.732+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:18.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:18.067+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:18.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:17.914+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:17.915582+0000) 2024-02-19T20:18:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:18.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:18.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:18.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:18.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:18.131+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:18.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:18.667+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.668501+0000) 2024-02-19T20:18:18.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:18.444+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.445645+0000) 2024-02-19T20:18:19.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:18.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:19.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:18.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:19.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:18.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:19.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:18.767+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:19.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:18.916+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:18.917682+0000) 2024-02-19T20:18:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:19.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:19.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:19.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:19.087+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:19.087+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:19.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:19.447+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.448270+0000) 2024-02-19T20:18:19.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:19.670+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.671173+0000) 2024-02-19T20:18:20.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:19.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:20.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:19.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:20.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:19.745+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:20.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:19.746+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:20.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:19.918+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:19.919727+0000) 2024-02-19T20:18:20.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:20.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:20.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:20.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:20.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:20.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:20.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:20.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:20.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:20.079+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:20.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:20.672+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.673249+0000) 2024-02-19T20:18:20.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:20.450+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.451032+0000) 2024-02-19T20:18:21.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:20.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:21.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:20.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:21.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:20.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:21.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:20.729+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:21.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:20.921+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:20.922273+0000) 2024-02-19T20:18:21.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:21.144+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:21.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:21.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:21.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:21.040+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:21.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:21.673+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.674234+0000) 2024-02-19T20:18:21.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:21.450+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.451932+0000) 2024-02-19T20:18:22.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:21.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:22.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:21.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:22.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:21.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:22.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:21.737+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:22.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:21.921+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:21.922957+0000) 2024-02-19T20:18:22.289 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:22.187+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.188458+0000) 2024-02-19T20:18:22.289 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:22.190+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59378 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:22.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:22.162+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:22.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:22.188+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.189282+0000) 2024-02-19T20:18:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:22.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:22.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:22.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:22.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:22.072+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:22.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:22.187+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.188898+0000) 2024-02-19T20:18:22.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:22.451+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.452106+0000) 2024-02-19T20:18:22.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:22.673+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.674441+0000) 2024-02-19T20:18:22.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:22.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:22.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:22.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:22.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:22.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:22.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:22.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:23.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:22.922+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:22.923153+0000) 2024-02-19T20:18:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:23.050+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:23.050+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:23.050+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:23.051+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:23.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:23.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:23.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:23.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:23.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:23.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:23.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:23.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:23.675+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.676495+0000) 2024-02-19T20:18:23.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:23.453+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.454103+0000) 2024-02-19T20:18:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:23.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:23.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:23.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:24.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:23.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:24.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:23.924+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:23.925557+0000) 2024-02-19T20:18:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:24.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:24.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:24.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:24.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:24.015+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:24.015+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:24.015+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:24.015+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:24.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:24.455+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.456295+0000) 2024-02-19T20:18:24.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:24.677+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.678711+0000) 2024-02-19T20:18:25.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:24.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:25.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:24.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:25.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:24.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:25.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:24.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:25.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:24.926+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:24.927533+0000) 2024-02-19T20:18:25.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:25.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:25.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:25.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:25.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:25.043+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:25.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:25.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:25.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:25.121+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:25.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:25.679+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.680966+0000) 2024-02-19T20:18:25.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:25.457+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.458480+0000) 2024-02-19T20:18:25.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:25.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:25.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:25.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:25.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:25.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:25.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:25.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:26.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:25.929+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:25.931010+0000) 2024-02-19T20:18:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:26.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:26.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:26.126+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:26.127+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:26.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:26.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:26.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:26.059+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:26.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:26.458+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.459188+0000) 2024-02-19T20:18:26.712 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:26.680+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.681822+0000) 2024-02-19T20:18:27.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:26.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:27.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:26.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:27.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:26.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:27.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:26.711+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:27.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:26.930+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:26.931718+0000) 2024-02-19T20:18:27.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:27.198+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59419 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:27.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:27.194+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.195413+0000) 2024-02-19T20:18:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:27.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:27.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:27.169+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:27.169+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:27.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:27.195+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.196206+0000) 2024-02-19T20:18:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:27.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:27.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:27.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:27.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:27.099+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:27.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:27.194+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.195273+0000) 2024-02-19T20:18:27.600 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:27.458+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.459351+0000) 2024-02-19T20:18:27.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:27.680+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.681983+0000) 2024-02-19T20:18:27.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:27.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:27.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:27.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:27.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:27.725+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:27.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:27.726+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:28.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:27.930+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:27.931886+0000) 2024-02-19T20:18:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:28.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:28.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:28.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:28.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:28.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:28.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:28.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:28.136+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:28.723 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:28.683+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.684571+0000) 2024-02-19T20:18:28.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:28.461+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.462624+0000) 2024-02-19T20:18:28.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:28.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:28.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:28.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:28.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:28.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:28.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:28.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:29.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:28.933+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:28.934360+0000) 2024-02-19T20:18:29.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:29.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:29.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:29.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:29.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:29.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:29.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:29.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:29.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:29.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:29.114+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:29.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:29.686+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.687096+0000) 2024-02-19T20:18:29.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:29.464+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.465067+0000) 2024-02-19T20:18:30.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:29.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:30.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:29.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:30.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:29.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:30.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:29.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:30.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:29.935+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:29.936795+0000) 2024-02-19T20:18:30.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:30.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:30.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:30.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:30.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:30.158+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:30.158+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:30.158+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:30.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:30.159+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:30.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:30.688+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.689839+0000) 2024-02-19T20:18:30.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:30.466+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.467743+0000) 2024-02-19T20:18:31.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:30.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:31.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:30.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:31.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:30.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:31.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:30.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:31.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:30.937+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:30.938947+0000) 2024-02-19T20:18:31.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:31.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:31.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:31.155+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:31.155+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:31.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:31.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:31.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:31.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:31.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:31.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:31.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:31.689+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.690612+0000) 2024-02-19T20:18:31.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:31.467+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.468778+0000) 2024-02-19T20:18:32.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:31.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:32.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:31.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:32.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:31.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:32.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:31.748+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:32.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:31.939+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:31.940036+0000) 2024-02-19T20:18:32.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:32.205+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59458 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:32.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:32.203+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.204153+0000) 2024-02-19T20:18:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:32.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:32.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:32.134+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:32.135+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:32.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:32.202+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.203798+0000) 2024-02-19T20:18:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:32.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:32.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:32.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:32.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:32.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:32.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:32.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:32.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:32.202+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.203853+0000) 2024-02-19T20:18:32.605 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:32.467+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.468984+0000) 2024-02-19T20:18:32.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:32.689+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.690824+0000) 2024-02-19T20:18:32.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:32.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:32.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:32.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:32.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:32.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:32.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:32.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:33.207 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:32.939+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:32.940202+0000) 2024-02-19T20:18:33.207 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:33.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:33.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:33.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:33.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:33.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:33.208 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:33.122+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:33.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:33.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:33.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:33.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:33.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:33.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:33.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:33.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:33.724 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:33.692+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.693117+0000) 2024-02-19T20:18:33.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:33.470+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.471062+0000) 2024-02-19T20:18:34.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:33.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:34.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:33.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:34.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:33.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:34.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:33.806+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:34.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:33.941+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:33.942568+0000) 2024-02-19T20:18:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:34.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:34.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:34.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:34.165+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:34.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:34.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:34.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:34.174+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:34.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:34.472+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.473276+0000) 2024-02-19T20:18:34.725 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:34.694+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.695229+0000) 2024-02-19T20:18:35.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:34.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:35.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:34.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:35.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:34.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:35.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:34.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:35.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:34.943+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:34.944793+0000) 2024-02-19T20:18:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:35.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:35.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:35.172+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:35.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:35.173+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:35.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:35.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:35.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:35.141+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:35.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:35.696+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.697919+0000) 2024-02-19T20:18:35.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:35.474+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.475344+0000) 2024-02-19T20:18:36.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:35.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:36.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:35.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:36.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:35.806+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:36.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:35.807+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:36.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:35.946+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:35.947054+0000) 2024-02-19T20:18:36.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:36.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:36.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:36.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:36.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:36.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:36.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:36.171+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:36.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:36.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:36.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:36.196+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:36.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:36.697+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.698744+0000) 2024-02-19T20:18:36.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:36.475+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.476186+0000) 2024-02-19T20:18:37.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:36.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:37.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:36.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:37.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:36.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:37.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:36.849+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:37.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:36.947+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:36.948136+0000) 2024-02-19T20:18:37.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:37.213+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59498 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:37.210+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.211095+0000) 2024-02-19T20:18:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:37.210+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.211095+0000) 2024-02-19T20:18:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:37.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:37.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:37.213+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:37.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:37.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:37.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:37.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:37.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:37.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:37.169+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:37.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:37.210+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.211736+0000) 2024-02-19T20:18:37.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:37.475+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.476350+0000) 2024-02-19T20:18:37.863 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:37.697+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.698995+0000) 2024-02-19T20:18:38.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:37.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:38.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:37.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:38.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:37.862+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:38.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:37.862+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:38.174 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:37.947+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:37.948283+0000) 2024-02-19T20:18:38.478 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:38.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:38.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:38.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:38.224+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:38.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:38.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:38.479 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:38.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:38.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:38.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:38.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:38.700+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.701573+0000) 2024-02-19T20:18:38.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:38.477+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.478316+0000) 2024-02-19T20:18:39.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:38.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:39.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:38.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:39.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:38.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:39.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:38.827+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:39.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:38.949+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:38.950604+0000) 2024-02-19T20:18:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:39.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:39.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:39.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:39.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:39.220+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:39.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:39.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:39.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:39.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:39.151+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:39.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:39.480+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.481972+0000) 2024-02-19T20:18:39.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:39.704+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.705067+0000) 2024-02-19T20:18:40.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:39.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:40.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:39.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:40.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:39.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:40.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:39.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:40.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:39.952+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:39.953053+0000) 2024-02-19T20:18:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:40.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:40.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:40.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:40.229+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:40.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:40.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:40.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:40.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:40.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:40.707+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.708580+0000) 2024-02-19T20:18:40.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:40.483+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.484406+0000) 2024-02-19T20:18:41.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:40.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:41.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:40.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:41.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:40.873+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:41.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:40.874+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:41.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:40.955+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:40.956513+0000) 2024-02-19T20:18:41.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:41.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:41.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:41.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:41.202+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:41.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:41.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:41.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:41.187+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:41.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:41.708+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.709706+0000) 2024-02-19T20:18:41.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:41.484+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.485140+0000) 2024-02-19T20:18:42.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:41.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:42.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:41.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:42.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:41.921+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:42.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:41.922+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:42.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:41.957+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:41.957800+0000) 2024-02-19T20:18:42.292 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:42.220+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59537 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:42.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:42.217+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.218614+0000) 2024-02-19T20:18:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:42.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:42.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:42.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:42.172+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:42.217+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.218393+0000) 2024-02-19T20:18:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:42.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:42.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:42.188+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:42.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:42.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:42.216+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.217846+0000) 2024-02-19T20:18:42.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:42.484+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.485337+0000) 2024-02-19T20:18:42.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:42.709+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.709913+0000) 2024-02-19T20:18:43.142 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:42.878+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:43.143 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:42.878+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:43.143 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:42.878+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:43.143 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:42.878+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:43.143 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:42.958+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:42.958045+0000) 2024-02-19T20:18:43.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:43.142+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:43.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:43.142+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:43.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:43.142+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:43.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:43.143+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:43.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:43.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:43.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:43.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:43.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:43.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:43.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:43.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:43.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:43.711+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.712354+0000) 2024-02-19T20:18:43.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:43.487+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.487591+0000) 2024-02-19T20:18:44.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:43.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:44.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:43.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:44.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:43.911+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:44.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:43.911+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:44.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:43.960+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:43.961582+0000) 2024-02-19T20:18:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:44.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:44.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:44.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:44.109+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:44.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:44.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:44.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:44.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:44.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:44.713+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.714369+0000) 2024-02-19T20:18:44.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:44.490+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.491159+0000) 2024-02-19T20:18:45.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:44.875+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:45.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:44.875+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:45.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:44.875+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:45.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:44.876+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:45.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:44.962+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:44.963615+0000) 2024-02-19T20:18:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:45.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:45.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:45.228+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:45.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:45.229+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:45.127+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:45.127+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:45.127+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:45.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:45.128+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:45.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:45.492+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.493922+0000) 2024-02-19T20:18:45.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:45.715+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.716884+0000) 2024-02-19T20:18:46.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:45.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:46.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:45.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:46.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:45.834+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:46.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:45.835+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:46.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:45.964+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:45.965691+0000) 2024-02-19T20:18:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:46.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:46.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:46.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:46.083+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:46.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:46.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:46.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:46.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:46.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:46.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:46.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:46.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:46.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:46.717+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.718064+0000) 2024-02-19T20:18:46.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:46.493+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.494750+0000) 2024-02-19T20:18:47.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:46.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:47.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:46.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:47.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:46.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:47.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:46.788+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:47.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:46.965+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:46.966489+0000) 2024-02-19T20:18:47.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:47.227+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59577 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:47.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:47.224+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.225967+0000) 2024-02-19T20:18:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:47.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:47.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:47.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:47.116+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:47.224+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.225531+0000) 2024-02-19T20:18:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:47.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:47.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:47.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:47.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:47.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:47.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:47.223+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.224784+0000) 2024-02-19T20:18:47.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:47.493+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.494990+0000) 2024-02-19T20:18:47.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:47.717+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.718236+0000) 2024-02-19T20:18:47.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:47.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:47.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:47.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:47.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:47.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:47.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:47.813+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:48.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:47.965+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:47.966684+0000) 2024-02-19T20:18:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:48.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:48.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:48.158+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:48.159+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:48.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:48.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:48.085+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:48.086+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:48.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:48.719+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.720349+0000) 2024-02-19T20:18:48.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:48.496+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.497048+0000) 2024-02-19T20:18:49.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:48.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:49.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:48.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:49.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:48.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:49.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:48.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:49.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:48.969+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:48.970247+0000) 2024-02-19T20:18:49.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:49.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:49.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:49.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:49.044+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:49.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:49.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:49.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:49.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:49.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:49.141+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:49.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:49.721+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.722376+0000) 2024-02-19T20:18:49.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:49.498+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.499390+0000) 2024-02-19T20:18:50.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:49.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:50.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:49.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:50.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:49.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:50.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:49.844+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:50.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:49.971+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:49.972890+0000) 2024-02-19T20:18:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:50.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:50.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:50.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:50.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:50.138+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:50.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:50.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:50.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:50.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:50.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:50.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:50.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:50.027+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:50.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:50.500+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.501375+0000) 2024-02-19T20:18:50.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:50.723+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.724500+0000) 2024-02-19T20:18:51.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:50.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:51.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:50.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:51.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:50.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:51.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:50.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:51.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:50.974+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:50.975363+0000) 2024-02-19T20:18:51.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:51.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:51.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:51.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:51.032+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:51.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:51.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:51.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:51.107+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:51.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:51.724+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.725699+0000) 2024-02-19T20:18:51.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:51.501+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.502566+0000) 2024-02-19T20:18:52.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:51.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:52.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:51.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:52.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:51.796+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:52.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:51.797+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:52.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:51.975+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:51.976044+0000) 2024-02-19T20:18:52.291 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:52.234+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59616 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:52.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:52.231+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.232760+0000) 2024-02-19T20:18:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:52.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:52.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:52.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:52.094+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:52.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:52.231+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.232883+0000) 2024-02-19T20:18:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:52.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:52.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:52.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:52.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:52.036+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:52.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:52.232+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.233145+0000) 2024-02-19T20:18:52.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:52.501+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.502731+0000) 2024-02-19T20:18:52.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:52.724+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.725915+0000) 2024-02-19T20:18:52.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:52.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:52.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:52.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:52.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:52.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:52.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:52.767+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:53.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:52.975+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:52.976214+0000) 2024-02-19T20:18:53.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.010+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:53.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.010+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:53.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.010+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:53.296 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.010+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:53.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:53.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:53.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:53.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:53.297 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:53.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:53.297 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:53.059+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:53.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:53.727+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.728288+0000) 2024-02-19T20:18:53.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.503+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.504880+0000) 2024-02-19T20:18:54.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:54.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:54.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.961+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:54.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:53.962+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:54.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:53.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:54.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:53.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:54.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:53.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:54.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:53.814+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:54.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:53.977+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:53.978059+0000) 2024-02-19T20:18:54.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:54.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:54.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:54.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:54.024+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:54.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:54.507+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.508076+0000) 2024-02-19T20:18:54.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:54.729+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.730341+0000) 2024-02-19T20:18:55.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:54.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:55.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:54.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:55.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:54.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:55.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:54.932+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:55.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:54.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:55.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:54.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:55.033 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:54.768+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:55.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:54.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:55.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:54.980+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:54.981578+0000) 2024-02-19T20:18:55.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:55.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:55.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:55.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:55.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:55.509+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.510271+0000) 2024-02-19T20:18:55.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.731+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.732375+0000) 2024-02-19T20:18:55.761 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:55.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:55.761 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:55.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:55.761 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:55.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:55.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:55.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:56.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:55.984+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:55.985204+0000) 2024-02-19T20:18:56.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:55.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:56.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:55.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:56.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:55.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:56.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:55.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:56.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:56.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:56.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:56.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:55.954+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:56.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:56.732+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.733124+0000) 2024-02-19T20:18:56.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:56.510+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.511317+0000) 2024-02-19T20:18:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:56.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:56.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:56.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:56.758+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:56.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:56.759+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:57.236 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:56.984+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:56.985936+0000) 2024-02-19T20:18:57.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:56.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:57.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:56.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:57.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:56.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:57.237 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:56.860+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:57.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:56.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:57.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:56.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:57.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:56.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:57.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:56.964+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:57.511 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:18:57.242+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59656 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:18:57.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:57.239+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.240068+0000) 2024-02-19T20:18:57.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:57.239+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.240438+0000) 2024-02-19T20:18:57.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:57.240+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.241414+0000) 2024-02-19T20:18:57.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:57.732+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.733297+0000) 2024-02-19T20:18:57.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:57.510+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.511525+0000) 2024-02-19T20:18:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:58.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:58.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:58.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:58.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:57.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:57.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:57.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:57.879+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:58.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:57.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:58.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:57.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:58.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:57.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:57.791+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:58.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:57.985+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:57.986133+0000) 2024-02-19T20:18:58.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:58.513+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.514127+0000) 2024-02-19T20:18:58.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:58.734+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.735488+0000) 2024-02-19T20:18:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:58.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:58.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:58.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:18:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:58.860+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:59.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:18:59.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.015+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:18:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:58.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:58.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:59.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:58.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:18:59.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:58.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:18:59.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:58.987+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:58.988143+0000) 2024-02-19T20:18:59.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.736+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.737524+0000) 2024-02-19T20:18:59.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:59.516+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.517578+0000) 2024-02-19T20:19:00.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:00.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:00.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:00.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:18:59.985+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:00.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:59.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:00.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:59.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:00.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:59.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:00.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:18:59.900+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:00.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:59.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:00.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:59.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:00.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:59.845+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:00.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:59.846+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:00.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:18:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:18:59.989+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:18:59.990039+0000) 2024-02-19T20:19:00.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:00.740+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.741166+0000) 2024-02-19T20:19:00.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:00.520+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.521201+0000) 2024-02-19T20:19:01.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:00.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:00.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:00.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:01.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:00.940+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:00.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:00.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:00.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:00.884+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:01.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:00.992+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:00.993709+0000) 2024-02-19T20:19:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:00.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:00.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:00.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:01.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:00.958+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:01.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:01.520+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.521962+0000) 2024-02-19T20:19:01.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:01.740+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.741893+0000) 2024-02-19T20:19:02.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:01.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:02.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:01.976+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:02.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:01.976+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:02.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:01.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:02.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:01.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:02.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:01.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:02.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:01.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:02.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:01.920+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:02.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:01.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:02.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:01.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:02.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:01.914+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:02.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:01.915+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:02.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:01.993+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:01.994947+0000) 2024-02-19T20:19:02.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:02.249+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59695 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:02.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:02.245+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.246601+0000) 2024-02-19T20:19:02.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:02.246+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.247907+0000) 2024-02-19T20:19:02.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:02.247+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.248195+0000) 2024-02-19T20:19:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:02.741+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.742127+0000) 2024-02-19T20:19:02.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:02.521+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.522158+0000) 2024-02-19T20:19:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:03.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:03.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:03.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:03.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:02.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:02.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:02.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:02.875+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:03.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:02.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:03.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:02.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:03.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:02.927+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:03.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:02.927+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:03.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:02.994+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:02.995124+0000) 2024-02-19T20:19:03.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:03.743+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.744163+0000) 2024-02-19T20:19:03.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:03.523+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.524359+0000) 2024-02-19T20:19:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:04.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:04.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:04.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:04.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:03.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:03.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:03.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:03.917+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:03.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:04.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:03.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:04.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:03.963+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:04.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:03.964+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:04.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:03.996+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:03.997209+0000) 2024-02-19T20:19:04.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:04.746+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.747702+0000) 2024-02-19T20:19:04.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:04.525+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.526502+0000) 2024-02-19T20:19:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:05.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:05.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:05.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:05.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:04.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:04.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:04.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:04.874+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:05.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:04.998+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:04.999435+0000) 2024-02-19T20:19:05.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:05.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:05.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:05.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:05.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:05.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:05.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:05.002+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:05.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:05.748+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.749902+0000) 2024-02-19T20:19:05.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:05.527+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:05.528515+0000) 2024-02-19T20:19:06.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:06.000+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.001721+0000) 2024-02-19T20:19:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:06.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:06.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:06.021+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:06.021+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:06.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:06.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:06.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:06.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:06.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:06.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:05.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:06.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:05.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:06.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:05.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:06.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:05.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:06.750+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.751128+0000) 2024-02-19T20:19:06.793 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:06.528+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:06.529272+0000) 2024-02-19T20:19:07.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:06.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:07.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:06.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:07.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:06.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:07.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:06.845+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:07.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:07.001+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.002488+0000) 2024-02-19T20:19:07.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:07.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:07.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:07.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:07.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:07.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:07.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:07.057+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:07.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:07.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:07.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:07.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:07.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:07.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:07.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:07.015+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:07.529 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:07.257+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59735 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:07.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:07.253+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.254263+0000) 2024-02-19T20:19:07.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:07.254+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.255980+0000) 2024-02-19T20:19:07.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:07.253+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.254423+0000) 2024-02-19T20:19:07.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:07.750+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.751292+0000) 2024-02-19T20:19:07.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:07.528+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:07.529460+0000) 2024-02-19T20:19:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:07.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:07.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:07.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:07.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:08.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:08.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:08.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:08.033+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:08.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:08.033+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:08.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:08.001+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.002672+0000) 2024-02-19T20:19:08.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:08.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:08.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:08.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:08.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:08.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:08.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:08.078+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:08.638 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:08.530+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.531692+0000) 2024-02-19T20:19:08.927 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:08.752+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:08.753427+0000) 2024-02-19T20:19:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:09.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:09.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:09.074+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:09.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:09.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:09.005+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.006125+0000) 2024-02-19T20:19:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:09.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:09.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:09.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:09.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:09.090+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:08.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:09.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:08.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:09.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:08.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:09.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:08.930+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:09.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:09.534+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.535130+0000) 2024-02-19T20:19:10.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:09.754+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:09.755634+0000) 2024-02-19T20:19:10.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:09.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:10.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:09.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:10.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:09.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:10.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:09.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:10.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:10.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:10.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:10.120+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:10.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:10.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:10.008+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.009685+0000) 2024-02-19T20:19:10.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:10.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:10.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:10.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:10.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:10.091+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:10.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:10.757+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.758148+0000) 2024-02-19T20:19:10.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:10.537+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:10.538643+0000) 2024-02-19T20:19:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:11.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:11.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:11.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:11.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:11.078+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:11.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:11.011+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.012526+0000) 2024-02-19T20:19:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:11.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:11.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:11.056+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:11.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:11.057+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:11.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:10.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:11.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:10.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:11.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:10.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:11.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:10.935+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:11.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:11.538+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.539432+0000) 2024-02-19T20:19:11.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:11.758+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:11.759058+0000) 2024-02-19T20:19:12.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:11.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:12.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:11.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:12.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:11.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:12.259 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:11.975+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:12.260 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:12.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:12.260 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:12.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:12.260 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:12.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:12.260 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:12.060+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:12.260 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:12.012+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.013268+0000) 2024-02-19T20:19:12.261 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:12.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:12.261 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:12.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:12.261 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:12.096+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:12.261 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:12.097+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:12.539 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:12.261+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.262929+0000) 2024-02-19T20:19:12.540 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:12.261+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.262554+0000) 2024-02-19T20:19:12.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:12.261+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.262468+0000) 2024-02-19T20:19:12.540 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:12.264+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59774 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:12.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:12.758+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.759262+0000) 2024-02-19T20:19:12.792 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:12.538+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:12.539650+0000) 2024-02-19T20:19:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:13.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:13.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:13.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:13.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:12.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:12.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:12.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:13.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:12.951+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:13.012+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.013448+0000) 2024-02-19T20:19:13.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:13.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:13.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:13.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:13.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:13.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:13.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:13.131+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:13.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:13.760+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.761303+0000) 2024-02-19T20:19:13.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:13.541+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:13.542238+0000) 2024-02-19T20:19:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:13.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:13.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:13.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:13.964+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:14.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:14.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:14.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:14.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:14.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:14.085+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:14.014+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.015458+0000) 2024-02-19T20:19:14.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:14.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:14.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:14.131+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:14.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:14.132+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:14.928 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:14.544+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.545726+0000) 2024-02-19T20:19:14.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:14.762+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:14.763961+0000) 2024-02-19T20:19:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:15.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:15.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:15.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:15.047+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:15.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:14.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:15.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:14.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:15.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:14.954+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:15.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:14.955+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:15.016+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.017552+0000) 2024-02-19T20:19:15.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:15.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:15.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:15.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:15.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:15.105+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:15.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:15.106+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:15.908 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:15.766+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.767633+0000) 2024-02-19T20:19:15.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:15.546+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:15.547731+0000) 2024-02-19T20:19:15.909 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:15.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:15.909 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:15.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:15.909 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:15.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:15.907+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:16.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:16.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:16.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:16.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:16.092+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:16.018+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.019630+0000) 2024-02-19T20:19:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:16.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:16.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:16.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:16.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:16.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:16.116+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:16.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:16.767+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.768539+0000) 2024-02-19T20:19:16.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:16.547+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:16.548841+0000) 2024-02-19T20:19:17.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:16.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:17.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:16.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:17.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:16.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:17.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:16.953+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:17.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:17.019+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.020423+0000) 2024-02-19T20:19:17.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:17.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:17.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:17.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:17.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:17.084+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:17.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:17.085+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:17.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:17.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:17.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:17.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:17.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:17.121+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:17.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:17.122+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:17.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:17.269+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.270568+0000) 2024-02-19T20:19:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:17.268+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.269958+0000) 2024-02-19T20:19:17.543 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:17.271+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59814 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:17.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:17.269+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.270889+0000) 2024-02-19T20:19:17.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:17.767+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.768772+0000) 2024-02-19T20:19:17.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:17.548+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:17.549041+0000) 2024-02-19T20:19:18.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:18.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:18.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:18.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:18.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:18.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:18.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:18.116+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:18.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:17.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:18.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:17.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:18.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:17.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:18.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:17.959+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:18.122 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:18.019+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.020599+0000) 2024-02-19T20:19:18.122 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:18.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:18.122 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:18.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:18.122 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:18.117+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:18.122 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:18.118+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:18.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:18.771+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.772336+0000) 2024-02-19T20:19:18.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:18.550+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:18.551098+0000) 2024-02-19T20:19:19.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:19.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:19.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:19.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:19.141+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:18.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:18.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:18.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:18.938+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:19.022+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.023175+0000) 2024-02-19T20:19:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:19.165+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:19.165+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:19.294 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:19.165+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:19.295 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:19.166+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:19.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:19.553+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.554678+0000) 2024-02-19T20:19:20.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:19.773+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:19.774542+0000) 2024-02-19T20:19:20.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:19.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:20.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:19.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:20.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:19.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:20.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:19.891+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:20.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:20.024+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.025179+0000) 2024-02-19T20:19:20.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:20.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:20.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:20.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:20.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:20.201+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:20.293 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:20.202+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:20.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:20.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:20.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:20.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:20.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:20.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:20.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:20.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:20.928 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:20.777+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.778202+0000) 2024-02-19T20:19:20.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:20.556+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:20.557252+0000) 2024-02-19T20:19:21.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:21.027+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.028611+0000) 2024-02-19T20:19:21.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:21.248+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:21.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:21.248+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:21.250 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:21.248+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:21.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:21.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:21.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:21.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:21.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:21.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:21.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:21.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:21.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:20.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:21.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:20.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:21.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:20.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:21.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:20.933+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:21.249+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:21.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:21.557+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.558076+0000) 2024-02-19T20:19:21.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:21.778+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:21.779509+0000) 2024-02-19T20:19:22.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:21.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:22.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:21.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:22.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:21.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:22.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:21.937+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:22.249 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:22.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:22.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:22.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:22.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:22.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:22.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:22.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:22.250 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:22.028+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.029888+0000) 2024-02-19T20:19:22.250 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:22.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:22.250 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:22.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:22.251 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:22.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:22.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:22.279+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59853 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:22.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:22.277+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.278074+0000) 2024-02-19T20:19:22.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:22.275+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.276901+0000) 2024-02-19T20:19:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:22.249+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:22.275+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.276551+0000) 2024-02-19T20:19:22.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:22.778+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.779693+0000) 2024-02-19T20:19:22.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:22.557+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:22.558259+0000) 2024-02-19T20:19:23.280 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:23.029+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.030103+0000) 2024-02-19T20:19:23.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:22.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:23.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:22.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:23.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:22.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:23.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:22.933+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:23.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:23.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:23.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:23.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:23.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:23.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:23.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:23.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:23.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:23.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:23.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:23.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:23.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:23.280+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:23.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:23.281+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:23.929 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:23.559+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.560429+0000) 2024-02-19T20:19:23.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:23.780+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:23.781691+0000) 2024-02-19T20:19:24.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:24.032+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.033541+0000) 2024-02-19T20:19:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:24.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:24.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:24.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:24.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:24.082+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:23.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:23.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:23.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:24.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:23.952+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:24.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:24.561+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.562487+0000) 2024-02-19T20:19:24.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:24.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:24.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:24.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:24.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:24.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:24.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:24.329+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:24.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:24.783+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:24.784127+0000) 2024-02-19T20:19:25.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:25.035+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.036111+0000) 2024-02-19T20:19:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:25.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:25.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:25.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:25.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:25.043+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:24.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:24.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:24.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:24.957+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:25.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:25.564+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.565123+0000) 2024-02-19T20:19:25.653 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:25.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:25.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:25.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:25.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:25.375+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:25.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:25.376+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:25.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:25.786+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:25.787595+0000) 2024-02-19T20:19:26.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:26.037+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.038144+0000) 2024-02-19T20:19:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:26.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:26.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:26.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:26.059+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:25.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:25.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:25.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:26.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:25.996+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:26.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:26.564+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.565907+0000) 2024-02-19T20:19:26.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:26.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:26.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:26.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:26.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:26.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:26.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:26.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:26.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:26.787+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:26.788432+0000) 2024-02-19T20:19:27.281 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:27.037+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.038933+0000) 2024-02-19T20:19:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:27.033+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:27.033+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:27.033+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:27.033+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:27.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:27.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:27.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:27.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.015+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:27.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:27.286+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59893 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.283+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.284268+0000) 2024-02-19T20:19:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:27.284+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.285157+0000) 2024-02-19T20:19:27.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:27.284+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.285255+0000) 2024-02-19T20:19:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:27.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:27.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:27.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:27.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:27.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.787+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.788624+0000) 2024-02-19T20:19:27.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:27.565+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:27.566059+0000) 2024-02-19T20:19:28.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:28.038+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.039127+0000) 2024-02-19T20:19:28.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:28.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:28.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:28.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:28.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:28.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:28.295 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:28.041+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:28.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:28.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:28.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:28.296 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:27.998+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:28.657 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:28.569+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.569671+0000) 2024-02-19T20:19:28.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:28.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:28.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:28.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:28.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:28.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:28.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:28.369+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:28.929 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:28.789+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:28.790709+0000) 2024-02-19T20:19:29.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:29.041+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.041197+0000) 2024-02-19T20:19:29.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:28.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:28.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:28.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:29.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:28.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:29.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:29.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:29.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:29.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:29.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:29.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:29.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:29.080+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:29.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:29.571+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.571870+0000) 2024-02-19T20:19:29.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:29.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:29.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:29.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:29.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:29.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:29.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:29.420+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:30.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:29.791+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:29.792933+0000) 2024-02-19T20:19:30.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:29.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:30.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:29.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:30.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:29.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:30.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:29.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:30.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:30.042+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.043219+0000) 2024-02-19T20:19:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:30.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:30.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:30.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:30.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:30.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:30.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:30.572+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.573982+0000) 2024-02-19T20:19:30.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:30.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:30.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:30.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:30.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:30.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:30.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:30.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:31.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:30.794+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:30.795058+0000) 2024-02-19T20:19:31.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:30.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:31.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:30.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:31.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:30.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:31.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:30.929+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:31.044+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.045291+0000) 2024-02-19T20:19:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:31.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:31.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:31.095+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:31.096+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:31.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:31.574+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.575190+0000) 2024-02-19T20:19:31.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:31.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:31.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:31.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:31.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:31.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:31.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:31.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:32.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:31.795+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:31.796080+0000) 2024-02-19T20:19:32.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:31.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:32.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:31.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:32.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:31.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:32.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:31.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:32.288 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:32.045+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.046072+0000) 2024-02-19T20:19:32.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:32.046+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:32.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:32.046+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:32.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:32.046+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:32.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:32.046+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:32.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:32.290+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.291575+0000) 2024-02-19T20:19:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:32.290+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.291504+0000) 2024-02-19T20:19:32.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:32.293+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59932 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:32.290+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.291597+0000) 2024-02-19T20:19:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:32.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:32.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:32.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:32.437+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:32.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:32.795+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.796260+0000) 2024-02-19T20:19:32.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:32.574+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:32.575381+0000) 2024-02-19T20:19:33.201 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:33.045+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.046246+0000) 2024-02-19T20:19:33.201 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:33.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:33.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:33.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:33.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:33.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:33.202 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:33.071+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:33.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:32.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:33.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:32.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:33.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:32.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:33.202 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:32.955+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:33.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:33.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:33.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:33.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:33.437+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:33.930 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:33.576+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.577526+0000) 2024-02-19T20:19:33.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:33.797+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:33.798286+0000) 2024-02-19T20:19:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:34.048+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.049704+0000) 2024-02-19T20:19:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:34.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:34.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:34.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:34.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:34.022+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:33.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:33.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:33.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:34.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:33.944+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:34.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:34.579+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.580064+0000) 2024-02-19T20:19:34.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:34.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:34.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:34.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:34.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:34.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:34.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:34.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:35.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:34.800+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:34.801858+0000) 2024-02-19T20:19:35.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:34.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:35.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:34.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:35.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:34.954+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:35.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:34.955+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:35.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:35.052+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.053175+0000) 2024-02-19T20:19:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:35.025+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:35.025+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:35.025+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:35.026+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:35.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:35.581+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.582133+0000) 2024-02-19T20:19:35.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:35.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:35.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:35.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:35.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:35.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:35.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:35.380+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:35.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:35.804+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:35.805508+0000) 2024-02-19T20:19:35.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:35.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:35.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:35.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:35.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:35.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:36.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:36.055+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.056679+0000) 2024-02-19T20:19:36.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:35.925+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:36.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:36.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:36.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:36.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:36.076+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:36.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:36.581+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.582964+0000) 2024-02-19T20:19:36.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:36.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:36.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:36.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:36.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:36.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:36.663 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:36.396+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:36.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:36.805+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:36.806354+0000) 2024-02-19T20:19:37.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:37.056+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.058007+0000) 2024-02-19T20:19:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:37.113+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:37.113+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:37.113+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:37.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:37.113+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:36.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:36.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:36.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:37.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:36.972+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:37.583 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:37.300+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 59972 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:37.583 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:37.297+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.298521+0000) 2024-02-19T20:19:37.584 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:37.298+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.299427+0000) 2024-02-19T20:19:37.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:37.297+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.298474+0000) 2024-02-19T20:19:37.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:37.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:37.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:37.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:37.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:37.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:37.585 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:37.355+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:37.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:37.805+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.806542+0000) 2024-02-19T20:19:37.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:37.582+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:37.583168+0000) 2024-02-19T20:19:38.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:38.057+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.058200+0000) 2024-02-19T20:19:38.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:38.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:38.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:38.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:38.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:38.131+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:38.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:38.131+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:38.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:37.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:38.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:37.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:38.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:37.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:38.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:37.944+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:38.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:38.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:38.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:38.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:38.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:38.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:38.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:38.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:38.909 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:38.585+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.586519+0000) 2024-02-19T20:19:38.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:38.809+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:38.810017+0000) 2024-02-19T20:19:38.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:38.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:38.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:38.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:38.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:38.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:39.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:39.059+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.060714+0000) 2024-02-19T20:19:39.181 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:38.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:39.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:39.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:39.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:39.499 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:39.365+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:39.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:39.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:39.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:39.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:39.180+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:39.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:39.588+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.589856+0000) 2024-02-19T20:19:40.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:39.811+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:39.812169+0000) 2024-02-19T20:19:40.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:39.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:40.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:39.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:40.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:39.879+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:40.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:39.880+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:40.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:40.061+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.062810+0000) 2024-02-19T20:19:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:40.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:40.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:40.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:40.166+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:40.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:40.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:40.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:40.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:40.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:40.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:40.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:40.318+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:40.888 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:40.592+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.593558+0000) 2024-02-19T20:19:40.888 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:40.813+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:40.814300+0000) 2024-02-19T20:19:40.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:40.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:40.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:40.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:40.889 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:40.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:41.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:40.888+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:41.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:41.063+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.064978+0000) 2024-02-19T20:19:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:41.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:41.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:41.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:41.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:41.131+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:41.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:41.593+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.594836+0000) 2024-02-19T20:19:41.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:41.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:41.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:41.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:41.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:41.354+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:41.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:41.356+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:41.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:41.814+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:41.815243+0000) 2024-02-19T20:19:41.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:41.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:41.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:41.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:41.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:41.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:41.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:41.896+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:42.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:42.064+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.065813+0000) 2024-02-19T20:19:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:42.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:42.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:42.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:42.098+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:42.595 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:42.307+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60011 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:42.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:42.306+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.307404+0000) 2024-02-19T20:19:42.595 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:42.304+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.305793+0000) 2024-02-19T20:19:42.596 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:42.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:42.596 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:42.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:42.596 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:42.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:42.596 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:42.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:42.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:42.304+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.305847+0000) 2024-02-19T20:19:42.873 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:42.594+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.595075+0000) 2024-02-19T20:19:42.873 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:42.814+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:42.815436+0000) 2024-02-19T20:19:42.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:42.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:42.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:42.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:42.874 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:42.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:43.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:43.065+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.066036+0000) 2024-02-19T20:19:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:42.873+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:43.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:43.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:43.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:43.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:43.082+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:43.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:43.596+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.597486+0000) 2024-02-19T20:19:43.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:43.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:43.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:43.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:43.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:43.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:43.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:43.355+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:43.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:43.816+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:43.817420+0000) 2024-02-19T20:19:43.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:43.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:43.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:43.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:43.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:43.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:43.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:43.854+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:44.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:44.067+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.068012+0000) 2024-02-19T20:19:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:44.033+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:44.033+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:44.033+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:44.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:44.034+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:44.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:44.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:44.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:44.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:44.600 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:44.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:44.601 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:44.321+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:44.897 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:44.599+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.600161+0000) 2024-02-19T20:19:44.897 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:44.819+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:44.820959+0000) 2024-02-19T20:19:44.897 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:44.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:44.897 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:44.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:44.897 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:44.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:45.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:45.070+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.071523+0000) 2024-02-19T20:19:45.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:44.897+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:45.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.003+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:45.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.003+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:45.602 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:45.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:45.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:45.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:45.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:45.313+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:45.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:45.314+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:45.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.601+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.602689+0000) 2024-02-19T20:19:45.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:45.823+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:45.824670+0000) 2024-02-19T20:19:45.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:45.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:45.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:45.854+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:45.856 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:45.854+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:46.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:46.072+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.073524+0000) 2024-02-19T20:19:46.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:45.854+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:46.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:45.983+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:46.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:46.347+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:46.604 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:46.347+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:46.604 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:46.347+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:46.604 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:46.348+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:46.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:46.602+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.603738+0000) 2024-02-19T20:19:46.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:46.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:46.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:46.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:46.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:46.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:46.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:46.817+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:46.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:46.824+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:46.825596+0000) 2024-02-19T20:19:47.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:47.073+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.074960+0000) 2024-02-19T20:19:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:47.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:47.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:47.006+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:47.007+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:47.604 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:47.315+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60050 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:47.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:47.311+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.312708+0000) 2024-02-19T20:19:47.605 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:47.312+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.313817+0000) 2024-02-19T20:19:47.605 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:47.312+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.313198+0000) 2024-02-19T20:19:47.606 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:47.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:47.606 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:47.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:47.606 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:47.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:47.606 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:47.337+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:47.931 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:47.602+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.603930+0000) 2024-02-19T20:19:47.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:47.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:47.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:47.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:47.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:47.810+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:47.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:47.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:47.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:47.824+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:47.825757+0000) 2024-02-19T20:19:48.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:48.074+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.075163+0000) 2024-02-19T20:19:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:48.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:48.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:48.055+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:48.055+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:48.607 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:48.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:48.607 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:48.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:48.607 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:48.340+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:48.607 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:48.340+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:48.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:48.606+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.607368+0000) 2024-02-19T20:19:48.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:48.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:48.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:48.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:48.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:48.778+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:48.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:48.779+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:48.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:48.827+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:48.828152+0000) 2024-02-19T20:19:49.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:49.076+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.077128+0000) 2024-02-19T20:19:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:49.067+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:49.067+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:49.067+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:49.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:49.067+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:49.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:49.609+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.611010+0000) 2024-02-19T20:19:49.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:49.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:49.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:49.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:49.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:49.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:49.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:49.365+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:49.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:49.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:49.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:49.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:49.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:49.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:49.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:49.810+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:49.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:49.829+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:49.830130+0000) 2024-02-19T20:19:50.219 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:50.078+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.079123+0000) 2024-02-19T20:19:50.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:50.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:50.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:50.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:50.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:50.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:50.220 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:50.087+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:50.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:50.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:50.371+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:50.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:50.372+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:50.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:50.613+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.614731+0000) 2024-02-19T20:19:50.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:50.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:50.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:50.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:50.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:50.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:50.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:50.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:50.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:50.831+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:50.832116+0000) 2024-02-19T20:19:51.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:51.080+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.081145+0000) 2024-02-19T20:19:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:51.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:51.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:51.069+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:51.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:51.069+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:51.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:51.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:51.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:51.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:51.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:51.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:51.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:51.347+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:51.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:51.615+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.616193+0000) 2024-02-19T20:19:51.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:51.831+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:51.832981+0000) 2024-02-19T20:19:51.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:51.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:51.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:51.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:51.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:51.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:51.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:51.847+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:52.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:52.080+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.081994+0000) 2024-02-19T20:19:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:52.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:52.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:52.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:52.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:52.071+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:52.616 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:52.322+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60090 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:52.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:52.319+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.320745+0000) 2024-02-19T20:19:52.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:52.319+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.320100+0000) 2024-02-19T20:19:52.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:52.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:52.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:52.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:52.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:52.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:52.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:52.365+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:52.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:52.319+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.320067+0000) 2024-02-19T20:19:52.866 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:52.615+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.616386+0000) 2024-02-19T20:19:52.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:52.832+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:52.833093+0000) 2024-02-19T20:19:52.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:52.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:52.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:52.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:52.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:52.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:53.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:53.081+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.082215+0000) 2024-02-19T20:19:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:52.866+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:53.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:53.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:53.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:53.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:53.063+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:53.676 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:53.618+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.619813+0000) 2024-02-19T20:19:53.676 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:53.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:53.677 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:53.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:53.677 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:53.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:53.677 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:53.406+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:53.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:53.835+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:53.836423+0000) 2024-02-19T20:19:53.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:53.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:53.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:53.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:53.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:53.855+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:53.933 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:53.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:54.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:54.083+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.084200+0000) 2024-02-19T20:19:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:54.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:54.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:54.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:54.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:54.099+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:54.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:54.621+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.622343+0000) 2024-02-19T20:19:54.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:54.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:54.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:54.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:54.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:54.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:54.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:54.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:55.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:54.838+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:54.839097+0000) 2024-02-19T20:19:55.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:54.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:55.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:54.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:55.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:54.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:55.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:54.849+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:55.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:55.085+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.086301+0000) 2024-02-19T20:19:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:55.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:55.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:55.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:55.100+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:55.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:55.624+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.625063+0000) 2024-02-19T20:19:55.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:55.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:55.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:55.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:55.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:55.456+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:55.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:55.457+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:56.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:55.840+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:55.841128+0000) 2024-02-19T20:19:56.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:55.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:56.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:55.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:56.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:55.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:56.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:55.896+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:56.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:56.087+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.088483+0000) 2024-02-19T20:19:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:56.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:56.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:56.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:56.071+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:56.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:56.624+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.625904+0000) 2024-02-19T20:19:56.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:56.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:56.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:56.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:56.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:56.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:56.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:56.482+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:57.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:56.841+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:56.842353+0000) 2024-02-19T20:19:57.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:56.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:57.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:56.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:57.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:56.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:57.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:56.916+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:57.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:57.088+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.089218+0000) 2024-02-19T20:19:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:57.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:57.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:57.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:57.120+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:57.626 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:19:57.329+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60129 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:19:57.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:57.327+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.328164+0000) 2024-02-19T20:19:57.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:57.328+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.329648+0000) 2024-02-19T20:19:57.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:57.326+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.327944+0000) 2024-02-19T20:19:57.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:57.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:57.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:57.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:57.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:57.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:57.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:57.521+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:57.886 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:57.625+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.626109+0000) 2024-02-19T20:19:57.886 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:57.841+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:57.842528+0000) 2024-02-19T20:19:57.886 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:57.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:57.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:57.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:57.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:57.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:58.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:58.088+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.089376+0000) 2024-02-19T20:19:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:57.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:58.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:58.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:58.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:58.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:58.129+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:58.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:58.627+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.628342+0000) 2024-02-19T20:19:58.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:58.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:58.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:58.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:58.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:58.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:58.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:58.486+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:59.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:58.844+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:58.845168+0000) 2024-02-19T20:19:59.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:58.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:59.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:58.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:59.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:58.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:19:59.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:58.849+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:19:59.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:59.090+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.091475+0000) 2024-02-19T20:19:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:59.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:59.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:59.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:59.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:19:59.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:59.109+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:19:59.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:19:59.629+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.630341+0000) 2024-02-19T20:19:59.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:59.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:59.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:59.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:59.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:59.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:19:59.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:19:59.528+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:00.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:59.846+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:19:59.847254+0000) 2024-02-19T20:20:00.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:59.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:00.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:59.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:00.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:59.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:00.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:19:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:19:59.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:00.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:00.092+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.093483+0000) 2024-02-19T20:20:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:00.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:00.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:00.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:00.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:00.096+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:00.849 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:00.631+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.632362+0000) 2024-02-19T20:20:00.849 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:00.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:00.849 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:00.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:00.849 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:00.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:00.849 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:00.566+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:01.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:01.094+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.095621+0000) 2024-02-19T20:20:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:00.849+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:00.850914+0000) 2024-02-19T20:20:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:00.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:00.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:00.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:01.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:00.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:01.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:01.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:01.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:01.068+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:01.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:01.632+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.633217+0000) 2024-02-19T20:20:01.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:01.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:01.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:01.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:01.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:01.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:01.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:01.610+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:01.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:01.851+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:01.852304+0000) 2024-02-19T20:20:01.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:01.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:01.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:01.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:01.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:01.851+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:01.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:01.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:02.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:02.096+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.097074+0000) 2024-02-19T20:20:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:02.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:02.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:02.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:02.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:02.107+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:02.577 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:02.337+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60169 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:02.578 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:02.334+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.335269+0000) 2024-02-19T20:20:02.578 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:02.333+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.334900+0000) 2024-02-19T20:20:02.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:02.334+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.335157+0000) 2024-02-19T20:20:02.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:02.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:02.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:02.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:02.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:02.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:02.852 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:02.632+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.633394+0000) 2024-02-19T20:20:02.853 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:02.577+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:02.853 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:02.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:02.853 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:02.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:02.853 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:02.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:02.853 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:02.815+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:03.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:03.096+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.097245+0000) 2024-02-19T20:20:03.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:02.851+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:02.852498+0000) 2024-02-19T20:20:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:03.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:03.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:03.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:03.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:03.092+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:03.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:03.634+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.635602+0000) 2024-02-19T20:20:03.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:03.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:03.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:03.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:03.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:03.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:03.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:03.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:03.856 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:03.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:03.856 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:03.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:03.856 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:03.775+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:03.856 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:03.776+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:03.853+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:03.854694+0000) 2024-02-19T20:20:04.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:04.099+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.100845+0000) 2024-02-19T20:20:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:04.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:04.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:04.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:04.101+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:04.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:04.636+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.637813+0000) 2024-02-19T20:20:04.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:04.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:04.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:04.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:04.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:04.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:04.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:04.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:04.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:04.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:04.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:04.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:04.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:04.800+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:04.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:04.801+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:04.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:04.856+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:04.857502+0000) 2024-02-19T20:20:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:05.102+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.103191+0000) 2024-02-19T20:20:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:05.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:05.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:05.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:05.106+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:05.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:05.638+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.639922+0000) 2024-02-19T20:20:05.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:05.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:05.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:05.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:05.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:05.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:05.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:05.629+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:05.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:05.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:05.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:05.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:05.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:05.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:05.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:05.752+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:05.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:05.859+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:05.860304+0000) 2024-02-19T20:20:06.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:06.104+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.105597+0000) 2024-02-19T20:20:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:06.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:06.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:06.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:06.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:06.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:06.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:06.104+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:06.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:06.639+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.640879+0000) 2024-02-19T20:20:06.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:06.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:06.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:06.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:06.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:06.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:06.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:06.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:06.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:06.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:06.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:06.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:06.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:06.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:06.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:06.747+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:06.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:06.861+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:06.862056+0000) 2024-02-19T20:20:07.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:07.105+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.106676+0000) 2024-02-19T20:20:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:07.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:07.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:07.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:07.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:07.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:07.104+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:07.609 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:07.344+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60208 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:07.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:07.341+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.342945+0000) 2024-02-19T20:20:07.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:07.341+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.342053+0000) 2024-02-19T20:20:07.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:07.341+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.342248+0000) 2024-02-19T20:20:07.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:07.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:07.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:07.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:07.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:07.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:07.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:07.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:07.862 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:07.640+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.641064+0000) 2024-02-19T20:20:07.863 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:07.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:07.863 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:07.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:07.863 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:07.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:07.863 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:07.698+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:07.863 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:07.861+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:07.862222+0000) 2024-02-19T20:20:08.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:08.105+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.106911+0000) 2024-02-19T20:20:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:08.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:08.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:08.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:08.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:08.075+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:08.865 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:08.642+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.643063+0000) 2024-02-19T20:20:08.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:08.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:08.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:08.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:08.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:08.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:08.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:08.585+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:08.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:08.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:08.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:08.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:08.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:08.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:08.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:08.682+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:09.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:09.108+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.109048+0000) 2024-02-19T20:20:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:08.864+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:08.865751+0000) 2024-02-19T20:20:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:09.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:09.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:09.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:09.032+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:09.768 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:09.644+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.645064+0000) 2024-02-19T20:20:09.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:09.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:09.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:09.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:09.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:09.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:09.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:09.603+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:09.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:09.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:09.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:09.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:09.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:09.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:09.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:09.683+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:10.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:09.867+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:09.868000+0000) 2024-02-19T20:20:10.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:10.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:10.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:10.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.015+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:10.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:10.110+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.111806+0000) 2024-02-19T20:20:10.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.646+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.647244+0000) 2024-02-19T20:20:10.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:10.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:10.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:10.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:10.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:10.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:10.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:10.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:10.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:10.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:10.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:10.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:10.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:10.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:10.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:10.712+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:10.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:10.869+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:10.870705+0000) 2024-02-19T20:20:11.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:11.113+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.114399+0000) 2024-02-19T20:20:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:10.979+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:11.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:11.647+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.648280+0000) 2024-02-19T20:20:11.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:11.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:11.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:11.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:11.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:11.689+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:11.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:11.690+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:11.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:11.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:11.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:11.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:11.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:11.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:11.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:11.762+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:11.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:11.870+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:11.871707+0000) 2024-02-19T20:20:12.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:12.114+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.115264+0000) 2024-02-19T20:20:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:11.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:11.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:11.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:11.941+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:12.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:12.351+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60248 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:12.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:12.348+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.349564+0000) 2024-02-19T20:20:12.649 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:12.350+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.351054+0000) 2024-02-19T20:20:12.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:12.349+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.350874+0000) 2024-02-19T20:20:12.934 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:12.647+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.648453+0000) 2024-02-19T20:20:12.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:12.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:12.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:12.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:12.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:12.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:12.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:12.664+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:12.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:12.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:12.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:12.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:12.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:12.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:12.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:12.721+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:12.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:12.871+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:12.871910+0000) 2024-02-19T20:20:13.265 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:13.115+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.115432+0000) 2024-02-19T20:20:13.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:12.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:13.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:12.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:13.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:12.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:13.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:12.959+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:13.933 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:13.650+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.651875+0000) 2024-02-19T20:20:13.933 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:13.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:13.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:13.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:13.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:13.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:13.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:13.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:13.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:13.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:13.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:13.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:13.934 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:13.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:13.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:13.746+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:13.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:13.873+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:13.874062+0000) 2024-02-19T20:20:14.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:14.116+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.117474+0000) 2024-02-19T20:20:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:13.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:13.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:13.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:13.933+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:14.876 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:14.653+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.654316+0000) 2024-02-19T20:20:14.876 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:14.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:14.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:14.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:14.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:14.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:14.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:14.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:14.877 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:14.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:14.877 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:14.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:14.877 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:14.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:14.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:14.724+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:14.878 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:14.876+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:14.876289+0000) 2024-02-19T20:20:15.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:15.118+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.119423+0000) 2024-02-19T20:20:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:14.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:14.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:14.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:14.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:15.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:15.656+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.656985+0000) 2024-02-19T20:20:15.880 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:15.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:15.880 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:15.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:15.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:15.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:15.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:15.572+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:15.881 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:15.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:15.881 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:15.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:15.881 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:15.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:15.881 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:15.766+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:15.879+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:15.879759+0000) 2024-02-19T20:20:16.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:16.122+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.123070+0000) 2024-02-19T20:20:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:15.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:15.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:15.940+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:16.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:15.940+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:16.880 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:16.657+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.658124+0000) 2024-02-19T20:20:16.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:16.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:16.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:16.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:16.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:16.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:16.881 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:16.541+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:16.882 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:16.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:16.882 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:16.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:16.882 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:16.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:16.882 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:16.771+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:16.882 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:16.879+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:16.880927+0000) 2024-02-19T20:20:17.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:17.123+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.124582+0000) 2024-02-19T20:20:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:16.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:16.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:16.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:17.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:16.969+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:17.658 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:17.359+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60283 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:17.658 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:17.355+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.356542+0000) 2024-02-19T20:20:17.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:17.355+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.356833+0000) 2024-02-19T20:20:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:17.356+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.357367+0000) 2024-02-19T20:20:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:17.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:17.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:17.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:17.659 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:17.552+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:17.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:17.657+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.658296+0000) 2024-02-19T20:20:17.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:17.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:17.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:17.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:17.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:17.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:17.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:17.778+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:17.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:17.880+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:17.881136+0000) 2024-02-19T20:20:18.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:18.123+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.124789+0000) 2024-02-19T20:20:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.000+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.000+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.000+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.001+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:18.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.659+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.660734+0000) 2024-02-19T20:20:18.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:18.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:18.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:18.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:18.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:18.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:18.885 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:18.542+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:18.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:18.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:18.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:18.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:18.886 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:18.736+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:18.886 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:18.738+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:19.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:19.125+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.126770+0000) 2024-02-19T20:20:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:18.883+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:18.884658+0000) 2024-02-19T20:20:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:19.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:19.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:18.976+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:19.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:19.661+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.662803+0000) 2024-02-19T20:20:19.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:19.523+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:19.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:19.523+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:19.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:19.523+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:19.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:19.524+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:19.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:19.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:19.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:19.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:19.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:19.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:19.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:19.699+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:20.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:19.886+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:19.887293+0000) 2024-02-19T20:20:20.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:20.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:20.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:20.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:20.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:20.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:20.015+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:20.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:20.127+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.128724+0000) 2024-02-19T20:20:20.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:20.663+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.664821+0000) 2024-02-19T20:20:20.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:20.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:20.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:20.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:20.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:20.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:20.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:20.513+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:20.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:20.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:20.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:20.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:20.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:20.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:20.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:20.740+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:21.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:20.889+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:20.890769+0000) 2024-02-19T20:20:21.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:21.129+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.130753+0000) 2024-02-19T20:20:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:21.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:21.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:21.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:21.048+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:21.892 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:21.664+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.665669+0000) 2024-02-19T20:20:21.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:21.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:21.892 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:21.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:21.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:21.724+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:21.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:21.725+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:21.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:21.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:21.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:21.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:21.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:21.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:21.893 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:21.551+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:22.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:22.130+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.131638+0000) 2024-02-19T20:20:22.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:21.891+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:21.892250+0000) 2024-02-19T20:20:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:22.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:22.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:22.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:22.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:22.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:22.081+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:22.665 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:22.366+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60323 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:22.666 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:22.363+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.364799+0000) 2024-02-19T20:20:22.666 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:22.362+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.363864+0000) 2024-02-19T20:20:22.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:22.363+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.364683+0000) 2024-02-19T20:20:22.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:22.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:22.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:22.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:22.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:22.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:22.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:22.507+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:22.935 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:22.664+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.665825+0000) 2024-02-19T20:20:22.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:22.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:22.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:22.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:22.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:22.761+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:22.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:22.762+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:22.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:22.891+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:22.892515+0000) 2024-02-19T20:20:23.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:23.130+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.131816+0000) 2024-02-19T20:20:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:23.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:23.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:23.058+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:23.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:23.058+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:23.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:23.668+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.669245+0000) 2024-02-19T20:20:23.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:23.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:23.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:23.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:23.771 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:23.514+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:23.771 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:23.515+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:23.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:23.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:23.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:23.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:23.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:23.716+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:23.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:23.717+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:23.893+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:23.894587+0000) 2024-02-19T20:20:24.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:24.132+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.133816+0000) 2024-02-19T20:20:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:24.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:24.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:24.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:24.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:24.082+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:24.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:24.671+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.672677+0000) 2024-02-19T20:20:24.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:24.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:24.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:24.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:24.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:24.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:24.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:24.474+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:24.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:24.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:24.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:24.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:24.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:24.682+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:24.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:24.683+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:25.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:24.895+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:24.896647+0000) 2024-02-19T20:20:25.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:25.135+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.136011+0000) 2024-02-19T20:20:25.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:25.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:25.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:25.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:25.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:25.099+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:25.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:25.099+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:25.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:25.675+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.676153+0000) 2024-02-19T20:20:25.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:25.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:25.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:25.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:25.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:25.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:25.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:25.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:26.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:25.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:25.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:25.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:25.712+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:26.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:25.897+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:25.898850+0000) 2024-02-19T20:20:26.409 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:26.137+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.138051+0000) 2024-02-19T20:20:26.409 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:26.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:26.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:26.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:26.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:26.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:26.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:26.128+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:26.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:26.129+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:26.677 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:26.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:26.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:26.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:26.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:26.409+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:26.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:26.676+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.677641+0000) 2024-02-19T20:20:26.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:26.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:26.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:26.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:26.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:26.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:26.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:26.747+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:26.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:26.899+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:26.900355+0000) 2024-02-19T20:20:27.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:27.137+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.138919+0000) 2024-02-19T20:20:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:27.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:27.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:27.098+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:27.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:27.098+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:27.677 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:27.373+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60362 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:27.678 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:27.370+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.371513+0000) 2024-02-19T20:20:27.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:27.370+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.371879+0000) 2024-02-19T20:20:27.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:27.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:27.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:27.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:27.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:27.364+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:27.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:27.364+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:27.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:27.370+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.371431+0000) 2024-02-19T20:20:27.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:27.676+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.677844+0000) 2024-02-19T20:20:27.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:27.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:27.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:27.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:27.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:27.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:27.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:27.780+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:27.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:27.899+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:27.900517+0000) 2024-02-19T20:20:28.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:28.138+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.139101+0000) 2024-02-19T20:20:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:28.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:28.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:28.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:28.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:28.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:28.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:28.125+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:28.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:28.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:28.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:28.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:28.681 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:28.381+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:28.681 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:28.381+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:28.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:28.679+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.680476+0000) 2024-02-19T20:20:28.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:28.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:28.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:28.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:28.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:28.818+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:28.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:28.819+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:28.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:28.901+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:28.902532+0000) 2024-02-19T20:20:29.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:29.140+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.141837+0000) 2024-02-19T20:20:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:29.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:29.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:29.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:29.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:29.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:29.091+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:29.682 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:29.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:29.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:29.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:29.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:29.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:29.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:29.384+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:29.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:29.681+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.682719+0000) 2024-02-19T20:20:29.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:29.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:29.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:29.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:29.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:29.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:29.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:29.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:29.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:29.903+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:29.904551+0000) 2024-02-19T20:20:30.220 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:30.143+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.144313+0000) 2024-02-19T20:20:30.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:30.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:30.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:30.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:30.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:30.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:30.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:30.094+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:30.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:30.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:30.407+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:30.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:30.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:30.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:30.684+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.685016+0000) 2024-02-19T20:20:30.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:30.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:30.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:30.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:30.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:30.880+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:30.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:30.880+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:30.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:30.907+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:30.908224+0000) 2024-02-19T20:20:31.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:31.145+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.146473+0000) 2024-02-19T20:20:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:31.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:31.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:31.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:31.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:31.140+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:31.686 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:31.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:31.686 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:31.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:31.686 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:31.365+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:31.687 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:31.365+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:31.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:31.685+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.686364+0000) 2024-02-19T20:20:31.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:31.908+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:31.909796+0000) 2024-02-19T20:20:31.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:31.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:31.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:31.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:31.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:31.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:31.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:31.913+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:32.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:32.146+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.147436+0000) 2024-02-19T20:20:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:32.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:32.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:32.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:32.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:32.130+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:32.686 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:32.380+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60402 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:32.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:32.377+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.378470+0000) 2024-02-19T20:20:32.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:32.378+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.379424+0000) 2024-02-19T20:20:32.687 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:32.378+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.379150+0000) 2024-02-19T20:20:32.687 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:32.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:32.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:32.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:32.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:32.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:32.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:32.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:32.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:32.685+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.686570+0000) 2024-02-19T20:20:32.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:32.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:32.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:32.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:32.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:32.890+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:32.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:32.890+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:32.938 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:32.908+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:32.909899+0000) 2024-02-19T20:20:33.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:33.146+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.147618+0000) 2024-02-19T20:20:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:33.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:33.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:33.108+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:33.109+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:33.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:33.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:33.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:33.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:33.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:33.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:33.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:33.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:34.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:33.688+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.689117+0000) 2024-02-19T20:20:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:33.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:33.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:33.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:33.885+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:34.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:33.911+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:33.912396+0000) 2024-02-19T20:20:34.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:34.149+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.151001+0000) 2024-02-19T20:20:34.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:34.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:34.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:34.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:34.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:34.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:34.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:34.083+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:34.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:34.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:34.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:34.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:34.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:34.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:34.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:34.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:35.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:34.690+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.691746+0000) 2024-02-19T20:20:35.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:34.913+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:34.914649+0000) 2024-02-19T20:20:35.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:34.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:35.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:34.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:35.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:34.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:35.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:34.921+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:35.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:35.153+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.154509+0000) 2024-02-19T20:20:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:35.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:35.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:35.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:35.038+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:35.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:35.692+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.693916+0000) 2024-02-19T20:20:35.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:35.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:35.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:35.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:35.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:35.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:35.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:35.455+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:35.916+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:35.917157+0000) 2024-02-19T20:20:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:35.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:35.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:35.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:35.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:36.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:36.155+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.156989+0000) 2024-02-19T20:20:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:36.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:36.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:36.024+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:36.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:36.024+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:36.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:36.694+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.695470+0000) 2024-02-19T20:20:36.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:36.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:36.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:36.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:36.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:36.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:36.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:36.459+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:37.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:36.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:36.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:36.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:36.897+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:37.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:36.917+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:36.918113+0000) 2024-02-19T20:20:37.383 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:37.156+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.157915+0000) 2024-02-19T20:20:37.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:37.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:37.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:37.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:37.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:37.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:37.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:37.069+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:37.695 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:37.387+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60441 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:37.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:37.385+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.386304+0000) 2024-02-19T20:20:37.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:37.385+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.386686+0000) 2024-02-19T20:20:37.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:37.384+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.385537+0000) 2024-02-19T20:20:37.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:37.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:37.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:37.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:37.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:38.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:37.694+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.695697+0000) 2024-02-19T20:20:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:37.917+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:37.918270+0000) 2024-02-19T20:20:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:37.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:37.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:38.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:37.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:38.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:37.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:38.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:38.157+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.158131+0000) 2024-02-19T20:20:38.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:38.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:38.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:38.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:38.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:38.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:38.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:38.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:38.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:38.057+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:38.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:38.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:38.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:38.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:38.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:38.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:39.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:38.697+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.698575+0000) 2024-02-19T20:20:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:38.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:38.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:38.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:38.895+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:38.920+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:38.921144+0000) 2024-02-19T20:20:39.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:39.159+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.160251+0000) 2024-02-19T20:20:39.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:39.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:39.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:39.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:39.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:39.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:39.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:39.083+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:39.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:39.699+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.700832+0000) 2024-02-19T20:20:39.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:39.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:39.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:39.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:39.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:39.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:39.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:39.473+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:40.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:39.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:39.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:39.856+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:39.857+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:39.923+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:39.924654+0000) 2024-02-19T20:20:40.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:40.161+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.162607+0000) 2024-02-19T20:20:40.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:40.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:40.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:40.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:40.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:40.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:40.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:40.124+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:40.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:40.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:40.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:40.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:40.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:40.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:40.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:40.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:41.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:40.701+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.702981+0000) 2024-02-19T20:20:41.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:40.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:40.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:40.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:40.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:41.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:40.925+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:40.926868+0000) 2024-02-19T20:20:41.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:41.163+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.164681+0000) 2024-02-19T20:20:41.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:41.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:41.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:41.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:41.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:41.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:41.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:41.082+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:41.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:41.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:41.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:41.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:41.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:41.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:41.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:41.420+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:42.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:41.703+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.704086+0000) 2024-02-19T20:20:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:41.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:41.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:41.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:41.916+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:42.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:41.926+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:41.927841+0000) 2024-02-19T20:20:42.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:42.164+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.165641+0000) 2024-02-19T20:20:42.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:42.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:42.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:42.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:42.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:42.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:42.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:42.041+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:42.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:42.391+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.392840+0000) 2024-02-19T20:20:42.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:42.393+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.394234+0000) 2024-02-19T20:20:42.705 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:42.395+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60481 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:42.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:42.391+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.392876+0000) 2024-02-19T20:20:42.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:42.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:42.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:42.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:42.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:42.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:42.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:42.425+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:43.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:42.703+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.704283+0000) 2024-02-19T20:20:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:42.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:42.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:42.904+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:42.905+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:43.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:42.927+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:42.928054+0000) 2024-02-19T20:20:43.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:43.164+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.165836+0000) 2024-02-19T20:20:43.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:43.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:43.437 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:43.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:43.437 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:43.090+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:43.437 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:43.090+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:43.707 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:43.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:43.707 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:43.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:43.707 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:43.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:43.707 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:43.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:44.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:43.706+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.707094+0000) 2024-02-19T20:20:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:43.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:43.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:43.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:44.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:43.898+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:44.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:43.929+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:43.930241+0000) 2024-02-19T20:20:44.421 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:44.166+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.167825+0000) 2024-02-19T20:20:44.421 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:44.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:44.421 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:44.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:44.421 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:44.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:44.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:44.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:44.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:44.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:44.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:44.048+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:44.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:44.048+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:44.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:44.420+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:45.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:44.708+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.709675+0000) 2024-02-19T20:20:45.020 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:45.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:45.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:44.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:45.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:44.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:45.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:44.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:45.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:44.895+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:45.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:44.931+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:44.932281+0000) 2024-02-19T20:20:45.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:45.170+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.171451+0000) 2024-02-19T20:20:45.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:45.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:45.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:45.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:45.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:45.019+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:45.712 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:45.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:45.712 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:45.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:45.712 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:45.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:45.712 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:45.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:46.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:45.711+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.712188+0000) 2024-02-19T20:20:46.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:45.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:46.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:45.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:46.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:45.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:46.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:45.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:46.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:45.933+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:45.934312+0000) 2024-02-19T20:20:46.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:46.172+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.173665+0000) 2024-02-19T20:20:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:46.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.036+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:46.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:46.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:46.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:46.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:46.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:46.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:46.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:46.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:46.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.712+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.713239+0000) 2024-02-19T20:20:46.992 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:46.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:46.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:46.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:46.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:46.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:46.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:46.889+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:46.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:46.934+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:46.935274+0000) 2024-02-19T20:20:47.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:47.173+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.174980+0000) 2024-02-19T20:20:47.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:47.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:46.992+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:47.713 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:47.402+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60521 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:47.713 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:47.399+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.400111+0000) 2024-02-19T20:20:47.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:47.399+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.400930+0000) 2024-02-19T20:20:47.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:47.401+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.402583+0000) 2024-02-19T20:20:47.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:47.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:47.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:47.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:47.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:47.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:47.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:47.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:48.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:47.712+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.713424+0000) 2024-02-19T20:20:48.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:47.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:48.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:47.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:48.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:47.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:48.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:47.895+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:48.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:47.934+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:47.935430+0000) 2024-02-19T20:20:48.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:48.174+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.175171+0000) 2024-02-19T20:20:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:48.036+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:48.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:48.036+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:48.036+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:48.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:48.037+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:48.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:48.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:48.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:48.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:48.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:48.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:48.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:48.390+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:49.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:48.714+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.715552+0000) 2024-02-19T20:20:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:48.937+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:48.938851+0000) 2024-02-19T20:20:49.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:48.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:48.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:48.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:49.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:48.943+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:49.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:49.176+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.177102+0000) 2024-02-19T20:20:49.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:49.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:49.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:49.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:49.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:49.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:49.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:49.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:49.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:49.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:49.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:49.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:49.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:49.079+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:49.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:49.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:50.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:49.718+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.719151+0000) 2024-02-19T20:20:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:49.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:49.932+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:49.932+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:49.932+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:49.940+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:49.941506+0000) 2024-02-19T20:20:50.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:50.178+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.179201+0000) 2024-02-19T20:20:50.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:50.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:50.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:50.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:50.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:50.406+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:50.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:50.072+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:50.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:50.072+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:50.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:50.072+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:50.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:50.072+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:50.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:50.407+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:51.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:50.720+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.721858+0000) 2024-02-19T20:20:51.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:51.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:51.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:51.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:51.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:51.032+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:51.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:50.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:51.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:50.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:51.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:50.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:51.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:50.927+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:51.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:50.942+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:50.943640+0000) 2024-02-19T20:20:51.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:51.033+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:51.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:51.180+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.181245+0000) 2024-02-19T20:20:51.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:51.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:51.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:51.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:51.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:51.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:51.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:51.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:51.722+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.723502+0000) 2024-02-19T20:20:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:51.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:51.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:51.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:52.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:51.928+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:52.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:51.943+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:51.944946+0000) 2024-02-19T20:20:52.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:52.181+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.182224+0000) 2024-02-19T20:20:52.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:52.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:52.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:52.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:52.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:52.061+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:52.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:52.061+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:52.719 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:52.409+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60561 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:52.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:52.406+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.407552+0000) 2024-02-19T20:20:52.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:52.406+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.407941+0000) 2024-02-19T20:20:52.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:52.408+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.409478+0000) 2024-02-19T20:20:52.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:52.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:52.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:52.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:52.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:52.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:52.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:52.466+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:53.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:52.722+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.723742+0000) 2024-02-19T20:20:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:52.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:52.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:52.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:52.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:53.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:52.944+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:52.945134+0000) 2024-02-19T20:20:53.428 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:53.181+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.182399+0000) 2024-02-19T20:20:53.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:53.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:53.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:53.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:53.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:53.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:53.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:53.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:53.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:53.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:53.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:53.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:53.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:53.083+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:53.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:53.428+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:53.725+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.726199+0000) 2024-02-19T20:20:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:53.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:53.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:54.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:53.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:54.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:53.939+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:54.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:53.946+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:53.947700+0000) 2024-02-19T20:20:54.414 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:54.183+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.184728+0000) 2024-02-19T20:20:54.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:54.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:54.415 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:54.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:54.415 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:54.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:54.415 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:54.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:54.415 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:54.117+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:54.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:54.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:54.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:54.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:54.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:54.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:54.727+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.728546+0000) 2024-02-19T20:20:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:54.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:54.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:55.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:54.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:55.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:54.922+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:55.044 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:54.949+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:54.950244+0000) 2024-02-19T20:20:55.411 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:55.185+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.186970+0000) 2024-02-19T20:20:55.411 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:55.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:55.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:55.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:55.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:55.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:55.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:55.130+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:55.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:55.130+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:55.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:55.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:55.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:55.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:55.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:55.411+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:56.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:55.731+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.732128+0000) 2024-02-19T20:20:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:55.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:55.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:56.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:55.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:56.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:55.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:56.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:55.951+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:55.952952+0000) 2024-02-19T20:20:56.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:56.187+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.188948+0000) 2024-02-19T20:20:56.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:56.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:56.402 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:56.137+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:56.402 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:56.137+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:56.402 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:56.137+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:56.402 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:56.137+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:56.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:56.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:56.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:56.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:56.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:56.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:57.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:56.732+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.733291+0000) 2024-02-19T20:20:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:56.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:56.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:56.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:56.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:57.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:56.952+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:56.953927+0000) 2024-02-19T20:20:57.411 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:57.188+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.190002+0000) 2024-02-19T20:20:57.411 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:57.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:57.411 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:57.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:57.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:57.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:57.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:57.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:57.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:57.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:57.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:57.104+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:57.720 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:20:57.417+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60600 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:20:57.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:57.415+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.416023+0000) 2024-02-19T20:20:57.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:57.414+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.415936+0000) 2024-02-19T20:20:57.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:57.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:57.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:57.411+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:57.723 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:57.415+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.416389+0000) 2024-02-19T20:20:58.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:57.732+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.733453+0000) 2024-02-19T20:20:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:57.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:57.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:57.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:57.885+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:58.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:57.953+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:57.954152+0000) 2024-02-19T20:20:58.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:58.189+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.190242+0000) 2024-02-19T20:20:58.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:58.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:58.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:58.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:58.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:58.102+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:58.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:58.103+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:58.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:58.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:58.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:58.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:58.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:58.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:58.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:58.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:59.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:58.734+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.735492+0000) 2024-02-19T20:20:59.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:58.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:59.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:58.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:59.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:58.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:20:59.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:58.858+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:20:59.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:58.955+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:58.956261+0000) 2024-02-19T20:20:59.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:59.191+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.192263+0000) 2024-02-19T20:20:59.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:59.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:59.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:59.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:59.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:59.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:59.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:59.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:59.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:59.145+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:20:59.430 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:59.146+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:20:59.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:59.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:20:59.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:20:59.429+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:00.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:20:59.736+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.738004+0000) 2024-02-19T20:21:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:59.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:59.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:59.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:00.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:59.882+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:00.043 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:20:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:20:59.958+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:20:59.959825+0000) 2024-02-19T20:21:00.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:00.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:00.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:00.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:00.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:00.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:00.383 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:00.185+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:00.384 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:00.193+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.194452+0000) 2024-02-19T20:21:00.384 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:00.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:00.384 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:00.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:00.384 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:00.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:00.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:00.382+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:01.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:00.740+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.741507+0000) 2024-02-19T20:21:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:00.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:00.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:00.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:00.896+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:01.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:00.961+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:00.962413+0000) 2024-02-19T20:21:01.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:01.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:01.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:01.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:01.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:01.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:01.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:01.146+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:01.348 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:01.195+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.196468+0000) 2024-02-19T20:21:01.348 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:01.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:01.348 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:01.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:01.348 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:01.346+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:01.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:01.346+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:02.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:01.742+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.742499+0000) 2024-02-19T20:21:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:01.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:01.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:01.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:01.934+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:02.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:01.963+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:01.963406+0000) 2024-02-19T20:21:02.392 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:02.197+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.197616+0000) 2024-02-19T20:21:02.393 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:02.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:02.393 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:02.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:02.393 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:02.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:02.393 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:02.194+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:02.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:02.426+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60640 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:02.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:02.422+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.422371+0000) 2024-02-19T20:21:02.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:02.422+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.422129+0000) 2024-02-19T20:21:02.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:02.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:02.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:02.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:02.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:02.392+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:02.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:02.393+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:02.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:02.423+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.423513+0000) 2024-02-19T20:21:03.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:02.741+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.742672+0000) 2024-02-19T20:21:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:02.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:02.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:02.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:02.942+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:03.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:02.962+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:02.963595+0000) 2024-02-19T20:21:03.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:03.196+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.197825+0000) 2024-02-19T20:21:03.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:03.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:03.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:03.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:03.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:03.150+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:03.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:03.150+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:03.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:03.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:03.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:03.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:03.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:03.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:03.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:03.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:03.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:03.964+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.965783+0000) 2024-02-19T20:21:03.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:03.745+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:03.746275+0000) 2024-02-19T20:21:04.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:04.198+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.199822+0000) 2024-02-19T20:21:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:04.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:04.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:04.187+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:04.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:04.188+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:03.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:03.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:03.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:03.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:04.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:04.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:04.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:04.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:04.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:04.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:04.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:04.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:05.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:04.747+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.748350+0000) 2024-02-19T20:21:05.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:04.966+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:04.967769+0000) 2024-02-19T20:21:05.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:05.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:05.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:05.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:05.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:05.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:05.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:05.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:05.202+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.203204+0000) 2024-02-19T20:21:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:05.144+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:05.144+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:05.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:05.144+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:05.145+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:05.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:05.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:05.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:05.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:05.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:05.428+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:05.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:05.429+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:06.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:05.749+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.750607+0000) 2024-02-19T20:21:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:05.969+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:05.970284+0000) 2024-02-19T20:21:06.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:06.204+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.205616+0000) 2024-02-19T20:21:06.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:06.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:06.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:06.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:06.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:06.124+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:06.456 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:06.124+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:06.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:06.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:06.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:06.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:06.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:06.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:06.456 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:06.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:06.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:06.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:06.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:06.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:06.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:06.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:06.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:06.455+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:06.970+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.971276+0000) 2024-02-19T20:21:07.047 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:06.750+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:06.751970+0000) 2024-02-19T20:21:07.427 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:07.206+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.207007+0000) 2024-02-19T20:21:07.427 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:07.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:07.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:07.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:07.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:07.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:07.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:07.078+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:07.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:07.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:07.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:07.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:07.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:07.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:07.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:07.087+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:07.721 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:07.432+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60679 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:07.721 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:07.429+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.430592+0000) 2024-02-19T20:21:07.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:07.429+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.430897+0000) 2024-02-19T20:21:07.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:07.430+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.431393+0000) 2024-02-19T20:21:07.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:07.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:07.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:07.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:07.723 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:07.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:07.723 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:07.496+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:07.970+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.971465+0000) 2024-02-19T20:21:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:07.751+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:07.752149+0000) 2024-02-19T20:21:08.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:08.206+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.207213+0000) 2024-02-19T20:21:08.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:08.100+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:08.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:08.100+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:08.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:08.100+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:08.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:08.100+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:08.523 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:08.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:08.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:08.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:08.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:08.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:08.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:08.084+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:08.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:08.753+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.754311+0000) 2024-02-19T20:21:08.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:08.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:08.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:08.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:08.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:08.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:08.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:08.522+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:09.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:08.973+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:08.974915+0000) 2024-02-19T20:21:09.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:09.208+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.209407+0000) 2024-02-19T20:21:09.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:09.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:09.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:09.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:09.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:09.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:09.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:09.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:09.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:09.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:09.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:09.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:09.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:09.079+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:09.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:09.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:09.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:09.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:09.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:09.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:09.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:09.512+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:09.977+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.978407+0000) 2024-02-19T20:21:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:09.755+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:09.756518+0000) 2024-02-19T20:21:10.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:10.210+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.211400+0000) 2024-02-19T20:21:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:10.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:10.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:10.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:10.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:10.079+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:10.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:10.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:10.041+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:10.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:10.041+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:10.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:10.759+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.760064+0000) 2024-02-19T20:21:10.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:10.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:10.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:10.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:10.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:10.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:10.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:10.549+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:11.213 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:11.214 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:11.214 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:11.214 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.016+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:11.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:10.980+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:10.981938+0000) 2024-02-19T20:21:11.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:11.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:11.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:11.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:11.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:11.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:11.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:11.091+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:11.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:11.212+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.213481+0000) 2024-02-19T20:21:11.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.760+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.761035+0000) 2024-02-19T20:21:11.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:11.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:11.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:11.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:11.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:11.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:11.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:11.566+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:12.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.995+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:12.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.995+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:12.220 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.995+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:12.220 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:11.996+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:12.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:11.982+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:11.983559+0000) 2024-02-19T20:21:12.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:12.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:12.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:12.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:12.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:12.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:12.221 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:12.085+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:12.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:12.436+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.437536+0000) 2024-02-19T20:21:12.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:12.440+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60719 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:12.213+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.214943+0000) 2024-02-19T20:21:12.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:12.437+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.438142+0000) 2024-02-19T20:21:12.543 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:12.436+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.437734+0000) 2024-02-19T20:21:12.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:12.760+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.761204+0000) 2024-02-19T20:21:12.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:12.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:12.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:12.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:12.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:12.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:12.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:12.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:13.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:13.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:13.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:13.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:13.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:13.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:13.215 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:13.004+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:13.215 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:12.982+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:12.983789+0000) 2024-02-19T20:21:13.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:13.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:13.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:13.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:13.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:13.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:13.216 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:13.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:13.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:13.214+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.215115+0000) 2024-02-19T20:21:13.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:13.763+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.764622+0000) 2024-02-19T20:21:13.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:13.555+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:13.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:13.555+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:13.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:13.555+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:13.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:13.556+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:14.218 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:14.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:14.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:14.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:14.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:14.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:14.219 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:14.022+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:14.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:13.984+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:13.985963+0000) 2024-02-19T20:21:14.219 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:14.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:14.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:14.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:14.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:14.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:14.220 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:14.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:14.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:14.217+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.218350+0000) 2024-02-19T20:21:14.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:14.767+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.768038+0000) 2024-02-19T20:21:14.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:14.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:14.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:14.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:14.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:14.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:14.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:14.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:15.221 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:15.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:15.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:15.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:15.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:15.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:15.222 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:15.036+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:15.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:14.987+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:14.988013+0000) 2024-02-19T20:21:15.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:15.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:15.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:15.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:15.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:15.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:15.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:15.093+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:15.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:15.220+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.221686+0000) 2024-02-19T20:21:15.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:15.770+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.771505+0000) 2024-02-19T20:21:15.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:15.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:15.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:15.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:15.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:15.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:15.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:15.585+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:16.225 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:16.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:16.225 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:16.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:16.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:16.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:16.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:16.012+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:16.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:15.989+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:15.990060+0000) 2024-02-19T20:21:16.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:16.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:16.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:16.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:16.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:16.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:16.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:16.056+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:16.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:16.224+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.225501+0000) 2024-02-19T20:21:16.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:16.771+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.772745+0000) 2024-02-19T20:21:16.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:16.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:16.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:16.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:16.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:16.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:16.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:16.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:17.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:17.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:17.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:17.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:17.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:17.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:17.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:17.019+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:17.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:16.990+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:16.991063+0000) 2024-02-19T20:21:17.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:17.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:17.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:17.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:17.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:17.078+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:17.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:17.079+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:17.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:17.444+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.445746+0000) 2024-02-19T20:21:17.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:17.447+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60758 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:17.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:17.444+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.445270+0000) 2024-02-19T20:21:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:17.225+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.226788+0000) 2024-02-19T20:21:17.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:17.444+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.445359+0000) 2024-02-19T20:21:17.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:17.771+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.772942+0000) 2024-02-19T20:21:17.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:17.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:17.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:17.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:17.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:17.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:17.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:17.609+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:18.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:18.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:18.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:18.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.027+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:18.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:17.990+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:17.991234+0000) 2024-02-19T20:21:18.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:18.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:18.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:18.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:18.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:18.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:18.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:18.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:18.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:18.226+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.227140+0000) 2024-02-19T20:21:18.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.775+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.776537+0000) 2024-02-19T20:21:18.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:18.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:18.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:18.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:18.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:18.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:18.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:18.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:19.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:19.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:19.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:19.230 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:18.984+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:19.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:18.992+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:18.993234+0000) 2024-02-19T20:21:19.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:19.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:19.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:19.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:19.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:19.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:19.230 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:19.063+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:19.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:19.228+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.229383+0000) 2024-02-19T20:21:19.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:19.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:19.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:19.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:19.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:19.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:19.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:19.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:20.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:19.994+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.995320+0000) 2024-02-19T20:21:20.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:20.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:19.779+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:19.780143+0000) 2024-02-19T20:21:20.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:19.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:20.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:19.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:20.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:19.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:20.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:19.949+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:20.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:20.230+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.231864+0000) 2024-02-19T20:21:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:20.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.020+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:20.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:20.781+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.782065+0000) 2024-02-19T20:21:20.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:20.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:20.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:20.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:20.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:20.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:20.943 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:20.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:21.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:20.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:21.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:20.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:21.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:20.947+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:21.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:20.948+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:21.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.996+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:20.997573+0000) 2024-02-19T20:21:21.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:21.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:21.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:21.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:20.999+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:21.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:21.233+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.234062+0000) 2024-02-19T20:21:21.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:21.782+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.783066+0000) 2024-02-19T20:21:21.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:21.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:21.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:21.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:21.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:21.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:21.903 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:21.606+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:22.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:21.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:22.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:21.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:22.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:21.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:22.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:21.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:22.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:21.997+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:21.998639+0000) 2024-02-19T20:21:22.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.026+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:22.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.026+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:22.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.026+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:22.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.027+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:22.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:22.451+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.452934+0000) 2024-02-19T20:21:22.542 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:22.455+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60798 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:22.234+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.235272+0000) 2024-02-19T20:21:22.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:22.452+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.453312+0000) 2024-02-19T20:21:22.543 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.452+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.453858+0000) 2024-02-19T20:21:22.854 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:22.782+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.783236+0000) 2024-02-19T20:21:22.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:22.563+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:22.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:22.563+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:22.854 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:22.563+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:22.855 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:22.564+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:23.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:22.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:23.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:22.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:23.236 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:22.853+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:23.236 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:22.853+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:23.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.997+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:22.998859+0000) 2024-02-19T20:21:23.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:23.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:23.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:23.237 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:22.999+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:23.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:23.234+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.235466+0000) 2024-02-19T20:21:23.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:23.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:23.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:23.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:23.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:23.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:23.873 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:23.594+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:23.873 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:23.784+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:23.785287+0000) 2024-02-19T20:21:23.873 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:23.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:23.873 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:23.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:23.874 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:23.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:24.239 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:23.872+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:24.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:24.000+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.001541+0000) 2024-02-19T20:21:24.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:24.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:24.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:24.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:24.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:24.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:24.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:24.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:24.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:24.238+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.239013+0000) 2024-02-19T20:21:24.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:24.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:24.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:24.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:24.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:24.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:24.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:24.593+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:24.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:24.786+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:24.787466+0000) 2024-02-19T20:21:24.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:24.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:24.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:24.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:24.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:24.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:25.242 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:24.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:25.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:25.002+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.003571+0000) 2024-02-19T20:21:25.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:25.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:25.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:25.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:25.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:25.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:25.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:25.036+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:25.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:25.241+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.242671+0000) 2024-02-19T20:21:25.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:25.790+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:25.791213+0000) 2024-02-19T20:21:25.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:25.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:25.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:25.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:25.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:25.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:25.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:25.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:25.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:25.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:25.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:25.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:25.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:25.631+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:26.244 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:25.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:26.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:26.005+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.006212+0000) 2024-02-19T20:21:26.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:26.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:26.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:26.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:26.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:26.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:26.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:26.045+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:26.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:26.244+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.245052+0000) 2024-02-19T20:21:26.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:26.791+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:26.792354+0000) 2024-02-19T20:21:26.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:26.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:26.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:26.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:26.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:26.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:26.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:26.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:26.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:26.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:27.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:26.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:27.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:26.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:27.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:26.886+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:27.246 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:27.006+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.007918+0000) 2024-02-19T20:21:27.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:27.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:27.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:27.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:27.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:27.046+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:27.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:27.046+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:27.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:27.462+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60837 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:27.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:27.459+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.460410+0000) 2024-02-19T20:21:27.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:27.458+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.459748+0000) 2024-02-19T20:21:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:27.245+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.246190+0000) 2024-02-19T20:21:27.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:27.458+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.459928+0000) 2024-02-19T20:21:27.888 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:27.791+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:27.792542+0000) 2024-02-19T20:21:27.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:27.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:27.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:27.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:27.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:27.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:27.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:27.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:28.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:27.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:28.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:27.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:28.246 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:27.887+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:28.247 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:27.888+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:28.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:28.007+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.008120+0000) 2024-02-19T20:21:28.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:28.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:28.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:28.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:28.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:28.031+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:28.247 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:28.032+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:28.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:28.245+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.246401+0000) 2024-02-19T20:21:28.869 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:28.794+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:28.795978+0000) 2024-02-19T20:21:28.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:28.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:28.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:28.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:28.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:28.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:28.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:28.563+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:29.248 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:28.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:29.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:28.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:29.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:28.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:29.249 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:28.869+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:29.249 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:29.009+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.010167+0000) 2024-02-19T20:21:29.249 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:29.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:29.249 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:29.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:29.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:29.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:29.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:29.022+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:29.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:29.247+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.248804+0000) 2024-02-19T20:21:29.828 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:29.798+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:29.799598+0000) 2024-02-19T20:21:29.828 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:29.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:29.829 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:29.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:29.829 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:29.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:29.829 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:29.566+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:30.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:29.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:30.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:29.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:30.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:29.827+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:30.224 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:29.827+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:30.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:30.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:30.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:30.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:30.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:30.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:30.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:30.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:30.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:30.011+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.012660+0000) 2024-02-19T20:21:30.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:30.250+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.251016+0000) 2024-02-19T20:21:30.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:30.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:30.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:30.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:30.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:30.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:30.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:30.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:30.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:30.779+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:30.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:30.779+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:30.802 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:30.779+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:30.803 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:30.780+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:31.252 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:30.800+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:30.801685+0000) 2024-02-19T20:21:31.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.013+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.014793+0000) 2024-02-19T20:21:31.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:31.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:31.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:31.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.020+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:31.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:31.252+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.253025+0000) 2024-02-19T20:21:31.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:31.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:31.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:31.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:31.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:31.545+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:31.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:31.546+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:31.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:31.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:31.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:31.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:31.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:31.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:31.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:31.768+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:32.254 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:31.802+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:31.803343+0000) 2024-02-19T20:21:32.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:32.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:32.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:32.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:31.984+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:32.255 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:32.015+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.016508+0000) 2024-02-19T20:21:32.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:32.469+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60877 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:32.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:32.466+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.467143+0000) 2024-02-19T20:21:32.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:32.466+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.467536+0000) 2024-02-19T20:21:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:32.253+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.254125+0000) 2024-02-19T20:21:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:32.468+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.469047+0000) 2024-02-19T20:21:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:32.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:32.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:32.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:32.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:32.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:32.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:32.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:32.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:32.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:32.937 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:32.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:32.938 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:32.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:32.938 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:32.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:32.938 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:32.768+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:32.938 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:32.769+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:32.939 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:32.802+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:32.803531+0000) 2024-02-19T20:21:33.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:32.936+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:33.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:33.015+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.016719+0000) 2024-02-19T20:21:33.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:33.253+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.254312+0000) 2024-02-19T20:21:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:33.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:33.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:33.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:33.542 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:33.476+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:34.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:33.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:34.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:33.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:34.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:33.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:34.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:33.802+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:34.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:33.805+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:33.806957+0000) 2024-02-19T20:21:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:33.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:33.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:33.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:33.929+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:34.020 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:34.017+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.018913+0000) 2024-02-19T20:21:34.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:34.256+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.257935+0000) 2024-02-19T20:21:34.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:34.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:34.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:34.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:34.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:34.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:34.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:34.438+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:35.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:34.809+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:34.810424+0000) 2024-02-19T20:21:35.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:34.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:35.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:34.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:35.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:34.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:35.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:34.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:35.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:34.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:35.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:34.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:35.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:34.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:35.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:34.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:35.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:35.258+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.259939+0000) 2024-02-19T20:21:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:35.020+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.021616+0000) 2024-02-19T20:21:35.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:35.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:35.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:35.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:35.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:35.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:35.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:35.480+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:36.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:35.813+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:35.814067+0000) 2024-02-19T20:21:36.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:35.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:36.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:35.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:36.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:35.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:36.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:35.858+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:36.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:35.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:35.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:35.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:35.968+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:36.022+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.023829+0000) 2024-02-19T20:21:36.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:36.262+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.263554+0000) 2024-02-19T20:21:36.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:36.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:36.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:36.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:36.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:36.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:36.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:36.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:37.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:36.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:37.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:36.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:37.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:36.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:37.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:36.964+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:37.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:36.814+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:36.815081+0000) 2024-02-19T20:21:37.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:36.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:37.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:36.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:37.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:36.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:37.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:36.881+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:37.292 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:37.263+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.264594+0000) 2024-02-19T20:21:37.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:37.024+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.025408+0000) 2024-02-19T20:21:37.727 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:37.477+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60916 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:37.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:37.473+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.474333+0000) 2024-02-19T20:21:37.729 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:37.474+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.475121+0000) 2024-02-19T20:21:37.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:37.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:37.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:37.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:37.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:37.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:37.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:37.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:37.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:37.473+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.474754+0000) 2024-02-19T20:21:38.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:37.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:38.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:37.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:38.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:37.926+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:38.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:37.926+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:38.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:37.814+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:37.815288+0000) 2024-02-19T20:21:38.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:37.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:38.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:37.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:38.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:37.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:38.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:37.837+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:38.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:38.263+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.264783+0000) 2024-02-19T20:21:38.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:38.024+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.025620+0000) 2024-02-19T20:21:38.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:38.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:38.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:38.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:38.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:38.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:38.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:38.518+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:39.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:38.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:39.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:38.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:39.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:38.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:39.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:38.939+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:39.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:38.816+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:38.817330+0000) 2024-02-19T20:21:39.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:38.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:39.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:38.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:39.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:38.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:39.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:38.839+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:39.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:39.266+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.267329+0000) 2024-02-19T20:21:39.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:39.027+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.028324+0000) 2024-02-19T20:21:39.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:39.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:39.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:39.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:39.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:39.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:39.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:39.513+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:40.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:39.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:40.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:39.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:40.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:39.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:40.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:39.948+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:40.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:39.818+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:39.819360+0000) 2024-02-19T20:21:40.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:39.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:40.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:39.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:40.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:39.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:40.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:39.836+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:40.291 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:40.268+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.269844+0000) 2024-02-19T20:21:40.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:40.029+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.030771+0000) 2024-02-19T20:21:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:40.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:40.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:40.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:40.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:41.273 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:40.821+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:40.822925+0000) 2024-02-19T20:21:41.273 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:40.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:41.274 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:40.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:41.274 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:40.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:41.274 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:40.837+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:41.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:40.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:41.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:40.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:41.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:40.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:41.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:40.987+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:41.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:41.033+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.034149+0000) 2024-02-19T20:21:41.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:41.272+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.273472+0000) 2024-02-19T20:21:41.828 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:41.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:41.828 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:41.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:41.828 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:41.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:41.828 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:41.577+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:41.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:41.823+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:41.824736+0000) 2024-02-19T20:21:41.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:41.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:41.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:41.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:41.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:41.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:41.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:41.827+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:42.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:41.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:42.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:41.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:42.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:41.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:42.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:41.965+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:42.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:42.034+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.035205+0000) 2024-02-19T20:21:42.541 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:42.484+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60956 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:42.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:42.481+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.482185+0000) 2024-02-19T20:21:42.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:42.481+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.482619+0000) 2024-02-19T20:21:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:42.274+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.275137+0000) 2024-02-19T20:21:42.543 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:42.481+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.482323+0000) 2024-02-19T20:21:42.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:42.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:42.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:42.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:42.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:42.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:42.945 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:42.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:42.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:42.823+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:42.824967+0000) 2024-02-19T20:21:42.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:42.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:42.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:42.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:42.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:42.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:42.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:42.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:43.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:42.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:43.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:42.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:43.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:42.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:43.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:42.956+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:43.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:43.034+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.035439+0000) 2024-02-19T20:21:43.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:43.274+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.275306+0000) 2024-02-19T20:21:43.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:43.827+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:43.828743+0000) 2024-02-19T20:21:43.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:43.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:43.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:43.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:43.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:43.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:43.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:43.589+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:44.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:43.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:44.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:43.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:44.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:43.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:44.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:43.842+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:43.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:43.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:43.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:43.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:44.037+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.038053+0000) 2024-02-19T20:21:44.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:44.276+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.277932+0000) 2024-02-19T20:21:44.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:44.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:44.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:44.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:44.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:44.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:44.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:44.585+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:44.829+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:44.830828+0000) 2024-02-19T20:21:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:44.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:44.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:44.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:44.942 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:44.833+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:45.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:44.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:45.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:44.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:45.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:44.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:45.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:44.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:45.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:45.039+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.040796+0000) 2024-02-19T20:21:45.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:45.279+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.280460+0000) 2024-02-19T20:21:45.833 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:45.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:45.833 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:45.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:45.833 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:45.550+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:45.833 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:45.551+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:46.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:45.832+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:45.833211+0000) 2024-02-19T20:21:46.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:45.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:46.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:45.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:46.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:45.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:46.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:45.880+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:46.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:45.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:46.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:45.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:46.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:45.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:46.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:45.922+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:46.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:46.042+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.043380+0000) 2024-02-19T20:21:46.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:46.283+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.283130+0000) 2024-02-19T20:21:46.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:46.833+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:46.834940+0000) 2024-02-19T20:21:46.914 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:46.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:46.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:46.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:46.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:46.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:46.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:46.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:46.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:46.594+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:47.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:46.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:47.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:46.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:47.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:46.914+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:47.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:46.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:47.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:46.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:47.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:46.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:47.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:46.967+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:47.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:47.044+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.044492+0000) 2024-02-19T20:21:47.544 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:47.488+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.489715+0000) 2024-02-19T20:21:47.545 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:47.490+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.491230+0000) 2024-02-19T20:21:47.545 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:47.492+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 60995 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:47.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:47.284+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.284744+0000) 2024-02-19T20:21:47.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:47.489+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.490643+0000) 2024-02-19T20:21:47.835 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:47.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:47.835 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:47.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:47.835 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:47.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:47.835 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:47.576+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:48.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:47.835+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:47.835168+0000) 2024-02-19T20:21:48.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:47.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:48.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:47.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:48.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:47.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:48.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:47.935+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:48.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:47.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:48.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:47.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:48.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:47.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:48.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:47.957+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:48.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:48.044+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.044769+0000) 2024-02-19T20:21:48.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:48.285+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.285094+0000) 2024-02-19T20:21:48.837 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:48.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:48.837 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:48.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:48.837 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:48.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:48.838 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:48.568+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:49.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:48.836+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:48.837256+0000) 2024-02-19T20:21:49.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:48.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:49.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:48.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:49.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:48.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:49.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:48.903+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:49.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:48.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:49.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:48.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:49.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:48.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:49.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:48.946+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:49.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:49.045+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.046812+0000) 2024-02-19T20:21:49.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:49.287+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.288059+0000) 2024-02-19T20:21:49.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:49.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:49.840 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:49.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:49.840 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:49.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:49.840 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:49.587+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:50.225 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:49.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:50.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:49.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:50.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:49.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:50.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:49.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:50.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:50.047+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.048880+0000) 2024-02-19T20:21:50.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:49.838+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:49.839784+0000) 2024-02-19T20:21:50.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:49.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:50.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:49.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:50.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:49.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:50.227 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:49.919+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:50.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:50.290+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.291548+0000) 2024-02-19T20:21:50.842 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:50.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:50.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:50.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:50.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:50.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:50.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:50.588+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:50.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:50.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:51.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:50.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:50.986+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:51.050+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.051181+0000) 2024-02-19T20:21:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:50.841+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:50.842283+0000) 2024-02-19T20:21:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:50.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:50.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:51.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:50.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:51.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:50.968+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:51.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:51.293+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.294205+0000) 2024-02-19T20:21:51.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:51.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:51.844 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:51.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:51.844 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:51.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:51.844 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:51.585+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:51.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:51.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:51.946+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:51.946+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:52.051+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.052351+0000) 2024-02-19T20:21:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:51.842+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:51.843853+0000) 2024-02-19T20:21:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:51.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:51.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:51.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:52.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:51.985+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:52.617 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:52.499+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61035 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:52.618 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:52.497+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.498455+0000) 2024-02-19T20:21:52.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:52.497+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.499001+0000) 2024-02-19T20:21:52.618 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:52.294+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.295301+0000) 2024-02-19T20:21:52.618 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:52.496+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.497247+0000) 2024-02-19T20:21:52.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:52.843+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:52.844102+0000) 2024-02-19T20:21:52.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:52.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:52.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:52.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:52.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:52.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:52.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:52.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:53.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:53.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:53.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:53.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:53.028+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:52.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:52.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:52.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:52.972+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:53.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:53.051+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.052536+0000) 2024-02-19T20:21:53.640 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:53.294+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.295487+0000) 2024-02-19T20:21:53.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:53.845+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:53.846133+0000) 2024-02-19T20:21:53.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:53.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:53.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:53.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:53.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:53.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:53.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:53.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:53.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:53.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:53.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:53.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:54.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:54.053+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.054656+0000) 2024-02-19T20:21:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:54.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:54.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:54.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:54.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:54.043+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:54.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:54.296+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.297654+0000) 2024-02-19T20:21:54.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:54.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:54.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:54.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:54.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:54.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:54.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:54.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:54.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:54.847+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:54.848343+0000) 2024-02-19T20:21:55.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:54.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:54.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:54.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:54.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:55.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:55.055+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.056686+0000) 2024-02-19T20:21:55.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:55.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:55.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:55.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:55.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:55.088+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:55.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:55.088+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:55.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:55.299+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.300405+0000) 2024-02-19T20:21:55.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:55.849+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:55.850399+0000) 2024-02-19T20:21:55.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:55.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:55.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:55.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:55.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:55.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:55.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:55.664+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:56.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:56.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:56.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:56.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:56.109+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:55.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:55.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:55.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:55.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:56.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:56.057+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.058888+0000) 2024-02-19T20:21:56.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:56.302+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.303045+0000) 2024-02-19T20:21:56.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:56.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:56.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:56.850+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:56.851522+0000) 2024-02-19T20:21:56.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:56.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:56.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:56.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:56.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:56.616+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:57.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:57.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:57.068+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:57.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:57.069+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:56.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:56.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:56.995+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:56.996+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:57.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:57.059+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.060046+0000) 2024-02-19T20:21:57.578 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:21:57.507+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61074 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:21:57.578 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:57.503+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.505010+0000) 2024-02-19T20:21:57.579 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:57.504+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.505232+0000) 2024-02-19T20:21:57.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:57.303+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.304866+0000) 2024-02-19T20:21:57.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:57.504+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.505669+0000) 2024-02-19T20:21:57.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:57.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:57.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:57.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:57.579 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:57.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:57.851 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:57.577+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:58.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:57.850+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:57.851813+0000) 2024-02-19T20:21:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:58.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:58.084+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:57.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:57.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:57.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:57.961+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:58.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:58.059+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.060245+0000) 2024-02-19T20:21:58.590 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:58.304+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.305072+0000) 2024-02-19T20:21:58.591 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:58.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:58.591 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:58.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:58.591 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:58.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:58.853 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:58.590+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:59.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:58.852+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:58.853872+0000) 2024-02-19T20:21:59.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:58.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:59.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:58.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:59.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:58.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:21:59.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:58.960+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:21:59.120 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:59.061+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.062663+0000) 2024-02-19T20:21:59.498 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:59.306+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.307109+0000) 2024-02-19T20:21:59.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:59.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:59.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:59.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:21:59.499 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:59.119+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:21:59.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:59.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:59.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:59.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:59.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:59.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:21:59.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:21:59.610+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:00.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:21:59.855+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:21:59.856032+0000) 2024-02-19T20:22:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:59.933+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:59.933+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:59.933+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:00.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:21:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:21:59.934+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:00.309 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:00.063+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.064865+0000) 2024-02-19T20:22:00.309 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:00.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:00.309 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:00.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:00.309 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:00.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:00.309 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:00.110+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:00.632 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:00.308+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.309146+0000) 2024-02-19T20:22:00.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:00.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:00.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:00.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:00.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:00.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:00.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:00.857+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:00.858049+0000) 2024-02-19T20:22:00.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:00.632+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:00.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:00.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:00.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:00.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:01.065+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.066894+0000) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:01.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:01.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:01.078+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:01.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:01.079+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:01.598 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:01.310+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.311255+0000) 2024-02-19T20:22:01.598 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:01.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:01.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:01.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:01.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:01.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:01.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:01.598+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:02.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:01.858+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:01.859138+0000) 2024-02-19T20:22:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:02.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:02.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:02.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:02.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:02.035+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:02.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:01.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:01.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:01.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:01.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:02.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:02.067+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.068014+0000) 2024-02-19T20:22:02.634 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:02.514+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61114 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:02.634 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:02.511+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.513000+0000) 2024-02-19T20:22:02.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:02.511+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.512648+0000) 2024-02-19T20:22:02.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:02.311+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.312344+0000) 2024-02-19T20:22:02.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:02.511+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.512193+0000) 2024-02-19T20:22:02.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:02.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:02.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:02.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:02.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:02.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:02.635 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:02.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:02.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:02.858+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:02.859368+0000) 2024-02-19T20:22:02.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:02.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:02.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:02.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:02.948 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:02.869+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:02.948 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:02.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:03.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:03.067+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.068251+0000) 2024-02-19T20:22:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:03.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:03.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:03.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:03.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:03.079+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:03.608 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:03.311+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.312552+0000) 2024-02-19T20:22:03.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:03.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:03.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:03.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:03.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:03.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:03.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:03.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:04.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:03.860+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:03.861342+0000) 2024-02-19T20:22:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:04.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:04.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:04.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:04.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:04.083+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:04.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:03.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:03.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:03.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:03.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:04.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:04.069+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.070428+0000) 2024-02-19T20:22:04.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:04.313+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.314761+0000) 2024-02-19T20:22:04.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:04.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:04.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:04.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:04.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:04.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:04.864 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:04.594+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:04.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:04.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:04.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:04.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:05.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:05.071+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.072446+0000) 2024-02-19T20:22:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:04.862+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:04.863615+0000) 2024-02-19T20:22:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:05.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:05.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:05.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:05.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:05.084+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:05.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:05.315+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.316950+0000) 2024-02-19T20:22:05.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:05.563+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:05.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:05.563+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:05.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:05.563+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:05.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:05.564+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:06.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:05.865+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:05.866126+0000) 2024-02-19T20:22:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:05.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:05.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:05.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:06.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:05.910+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:06.319 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:06.073+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.074518+0000) 2024-02-19T20:22:06.320 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:06.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:06.320 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:06.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:06.320 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:06.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:06.320 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:06.060+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:06.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:06.318+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.319616+0000) 2024-02-19T20:22:06.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:06.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:06.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:06.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:06.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:06.565+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:06.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:06.566+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:07.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:06.866+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:06.867490+0000) 2024-02-19T20:22:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:06.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:06.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:06.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:07.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:06.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:07.321 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:07.074+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.075693+0000) 2024-02-19T20:22:07.321 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:07.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:07.321 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:07.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:07.321 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:07.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:07.321 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:07.070+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:07.640 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:07.522+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61153 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:07.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:07.518+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.519676+0000) 2024-02-19T20:22:07.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:07.519+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.520097+0000) 2024-02-19T20:22:07.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:07.319+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.320881+0000) 2024-02-19T20:22:07.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:07.518+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.519355+0000) 2024-02-19T20:22:07.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:07.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:07.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:07.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:07.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:07.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:07.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:07.521+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:07.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:07.866+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:07.867700+0000) 2024-02-19T20:22:08.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:08.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:08.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:08.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:08.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:08.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:08.294 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:08.052+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:08.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:07.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:08.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:07.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:08.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:07.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:08.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:07.987+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:08.295 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:08.074+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.075890+0000) 2024-02-19T20:22:08.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:08.320+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.321099+0000) 2024-02-19T20:22:08.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:08.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:08.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:08.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:08.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:08.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:08.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:08.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:08.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:08.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:08.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:08.984+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:09.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:09.077+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.078048+0000) 2024-02-19T20:22:09.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:08.868+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:08.869866+0000) 2024-02-19T20:22:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:09.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:09.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:09.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:09.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:09.053+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:09.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:09.322+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.323746+0000) 2024-02-19T20:22:09.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:09.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:09.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:09.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:09.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:09.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:09.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:09.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:10.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:09.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:10.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:09.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:10.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:09.942+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:10.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:09.943+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:10.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:09.871+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:09.872339+0000) 2024-02-19T20:22:10.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:10.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:10.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:10.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:10.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:10.029+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:10.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:10.030+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:10.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:10.079+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.080063+0000) 2024-02-19T20:22:10.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:10.324+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.325853+0000) 2024-02-19T20:22:10.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:10.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:10.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:10.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:10.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:10.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:10.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:10.468+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:11.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:10.873+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:10.874905+0000) 2024-02-19T20:22:11.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:11.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:11.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:10.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:11.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:10.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:11.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:10.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:11.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:10.907+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:11.081+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.082235+0000) 2024-02-19T20:22:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:11.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:11.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:11.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:11.031+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:11.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:11.326+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.327858+0000) 2024-02-19T20:22:11.729 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:11.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:11.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:11.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:11.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:11.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:11.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:11.508+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:12.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:11.875+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:11.876063+0000) 2024-02-19T20:22:12.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:12.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:12.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:12.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:12.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:11.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:12.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:11.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:12.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:11.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:12.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:11.918+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:12.082+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.083343+0000) 2024-02-19T20:22:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:12.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:12.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:12.027+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:12.649 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:12.529+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61193 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:12.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:12.526+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.527581+0000) 2024-02-19T20:22:12.650 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:12.527+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.528547+0000) 2024-02-19T20:22:12.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:12.327+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.328984+0000) 2024-02-19T20:22:12.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:12.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:12.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:12.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:12.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:12.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:12.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:12.463+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:12.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:12.526+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.527856+0000) 2024-02-19T20:22:12.948 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:12.875+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:12.876237+0000) 2024-02-19T20:22:12.948 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:12.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:12.948 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:12.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:12.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:12.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:12.949 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:12.888+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:13.082+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.083539+0000) 2024-02-19T20:22:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:13.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.032+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:13.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:13.328+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.329201+0000) 2024-02-19T20:22:13.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:13.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:13.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:13.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:13.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:13.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:13.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:13.446+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:13.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.877+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:13.878621+0000) 2024-02-19T20:22:13.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:13.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:13.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:13.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:13.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:13.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:13.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:13.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:13.934+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:13.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:13.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:14.085+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.086225+0000) 2024-02-19T20:22:14.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:13.984+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:14.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:14.330+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.331227+0000) 2024-02-19T20:22:14.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:14.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:14.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:14.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:14.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:14.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:14.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:14.468+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:15.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:14.881+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:14.882054+0000) 2024-02-19T20:22:15.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:14.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:15.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:14.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:15.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:14.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:15.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:14.964+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:15.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:15.087+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.088292+0000) 2024-02-19T20:22:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:15.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.031+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:15.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:15.332+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.333295+0000) 2024-02-19T20:22:15.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:15.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:15.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:15.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:15.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:15.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:15.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:15.511+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:15.983 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.884+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:15.885468+0000) 2024-02-19T20:22:15.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:15.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:15.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.982+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:16.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:15.983+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:16.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.089+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.090503+0000) 2024-02-19T20:22:16.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:16.334+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.335409+0000) 2024-02-19T20:22:16.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:16.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:16.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:16.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:16.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:16.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:16.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:16.557+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:16.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:16.885+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:16.886746+0000) 2024-02-19T20:22:16.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:16.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:16.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:16.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:16.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:16.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:16.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:16.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:16.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:16.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:16.965+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:17.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:16.988+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:17.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:17.091+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.092359+0000) 2024-02-19T20:22:17.654 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:17.536+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61232 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:17.654 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:17.534+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.535828+0000) 2024-02-19T20:22:17.655 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:17.533+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.534916+0000) 2024-02-19T20:22:17.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:17.335+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.336492+0000) 2024-02-19T20:22:17.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:17.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:17.655 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:17.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:17.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:17.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:17.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:17.526+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:17.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:17.534+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.535372+0000) 2024-02-19T20:22:17.948 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:17.885+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:17.886914+0000) 2024-02-19T20:22:18.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.021+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:18.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.021+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:17.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:18.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:17.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:17.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:17.980+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:18.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:18.091+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.092587+0000) 2024-02-19T20:22:18.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:18.335+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.336695+0000) 2024-02-19T20:22:18.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:18.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:18.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:18.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:18.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:18.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:18.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:18.567+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:18.998 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.887+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:18.888986+0000) 2024-02-19T20:22:18.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:18.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:18.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.998+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:19.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:18.998+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:19.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:19.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.093+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.094791+0000) 2024-02-19T20:22:19.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:19.337+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.338674+0000) 2024-02-19T20:22:19.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:19.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:19.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:19.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:19.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:19.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:19.892 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:19.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:20.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:19.891+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:19.892372+0000) 2024-02-19T20:22:20.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:20.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:20.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:20.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:20.228 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:20.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:20.229 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:20.003+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:20.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:20.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:20.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.986+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:20.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:19.986+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:20.229 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:20.095+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.096919+0000) 2024-02-19T20:22:20.541 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:20.339+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.340699+0000) 2024-02-19T20:22:20.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:20.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:20.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:20.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:20.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:20.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:20.894 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:20.635+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:21.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:20.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:20.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:20.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:20.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:21.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:21.098+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.099396+0000) 2024-02-19T20:22:21.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:20.893+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:20.894473+0000) 2024-02-19T20:22:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:21.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.006+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:21.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:21.341+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.342836+0000) 2024-02-19T20:22:21.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:21.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:21.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:21.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:21.611 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:21.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:21.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:21.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.007+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.099+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.100845+0000) 2024-02-19T20:22:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.894+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:21.895966+0000) 2024-02-19T20:22:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:21.999+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:22.539 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:22.343+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.344133+0000) 2024-02-19T20:22:22.791 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:22.540+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.541941+0000) 2024-02-19T20:22:22.792 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:22.544+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61272 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:22.793 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.541+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.542625+0000) 2024-02-19T20:22:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:22.540+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.541762+0000) 2024-02-19T20:22:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:22.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:22.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:22.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:22.793 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:22.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:23.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:23.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:23.054 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:23.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:22.967+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:23.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:22.895+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:22.896190+0000) 2024-02-19T20:22:23.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.010+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:23.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.010+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:23.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.010+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:23.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.010+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:23.344 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:23.100+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.101038+0000) 2024-02-19T20:22:23.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:23.343+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.344348+0000) 2024-02-19T20:22:23.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:23.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:23.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:23.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:23.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:23.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:23.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:23.630+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:24.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:24.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:24.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:24.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:24.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:24.102+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.103029+0000) 2024-02-19T20:22:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.897+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:23.898215+0000) 2024-02-19T20:22:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:24.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:23.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:24.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:24.345+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.346441+0000) 2024-02-19T20:22:24.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:24.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:24.609 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:24.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:24.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:24.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:24.900 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:24.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:25.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:25.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:25.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:25.056+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:25.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:25.104+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.105058+0000) 2024-02-19T20:22:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:24.899+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:24.900331+0000) 2024-02-19T20:22:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:24.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:24.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:24.958+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:25.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:24.958+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:25.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:25.349+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.350108+0000) 2024-02-19T20:22:25.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:25.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:25.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:25.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:25.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:25.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:25.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:25.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:26.006 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:25.901+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:25.902495+0000) 2024-02-19T20:22:26.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:26.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:26.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:26.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:26.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:26.005+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:26.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:26.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:26.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:26.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:26.090+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:26.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:26.106+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.107079+0000) 2024-02-19T20:22:26.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:26.351+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.352887+0000) 2024-02-19T20:22:26.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:26.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:26.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:26.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:26.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:26.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:26.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:26.593+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:27.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:26.902+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:26.903703+0000) 2024-02-19T20:22:27.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.038+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:27.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.038+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:27.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.038+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:27.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.038+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:27.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:27.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:27.082+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:27.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:27.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:27.107+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.108404+0000) 2024-02-19T20:22:27.671 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:27.552+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61311 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:27.672 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:27.549+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.550273+0000) 2024-02-19T20:22:27.672 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.548+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.549485+0000) 2024-02-19T20:22:27.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:27.353+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.354171+0000) 2024-02-19T20:22:27.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:27.548+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.549743+0000) 2024-02-19T20:22:27.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:27.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:27.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:27.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:27.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:27.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:27.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:27.589+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:27.949 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.902+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:27.903986+0000) 2024-02-19T20:22:28.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:28.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:28.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:28.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:27.999+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:28.107+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.108575+0000) 2024-02-19T20:22:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:28.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:28.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:28.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:28.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:28.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:28.116+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:28.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:28.353+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.354349+0000) 2024-02-19T20:22:28.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:28.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:28.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:28.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:28.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:28.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:28.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:28.600+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:29.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:28.905+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:28.906045+0000) 2024-02-19T20:22:29.357 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:29.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:29.358 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:29.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:29.358 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:29.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:29.358 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:29.048+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:29.358 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:29.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:29.358 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:29.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:29.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:29.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:29.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:29.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:29.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:29.111+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.112013+0000) 2024-02-19T20:22:29.731 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:29.356+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.357790+0000) 2024-02-19T20:22:29.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:29.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:29.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:29.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:29.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:29.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:29.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:29.602+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:30.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:29.907+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:29.908174+0000) 2024-02-19T20:22:30.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:30.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:30.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:30.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:30.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:30.039+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:30.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:30.040+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:30.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:30.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:30.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:30.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:30.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:30.114+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.115481+0000) 2024-02-19T20:22:30.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:30.358+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.359923+0000) 2024-02-19T20:22:30.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:30.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:30.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:30.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:30.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:30.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:30.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:30.569+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:31.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:30.910+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:30.911650+0000) 2024-02-19T20:22:31.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:31.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:31.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:31.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:31.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:31.076+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:31.364 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:31.077+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:31.364 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:31.117+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.118779+0000) 2024-02-19T20:22:31.364 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:31.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:31.364 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:31.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:31.364 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:31.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:31.364 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:31.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:31.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:31.362+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.363443+0000) 2024-02-19T20:22:31.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:31.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:31.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:31.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:31.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:31.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:31.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:31.609+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:32.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:31.912+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:31.913582+0000) 2024-02-19T20:22:32.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:32.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:32.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:32.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:32.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:32.057+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:32.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:32.057+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:32.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:32.119+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.120629+0000) 2024-02-19T20:22:32.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:32.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:32.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:32.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:32.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:32.148+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:32.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:32.148+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:32.678 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:32.559+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61351 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:32.679 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:32.556+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.557617+0000) 2024-02-19T20:22:32.679 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:32.555+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.556047+0000) 2024-02-19T20:22:32.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:32.364+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.365257+0000) 2024-02-19T20:22:32.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:32.556+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.557767+0000) 2024-02-19T20:22:32.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:32.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:32.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:32.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:32.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:32.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:32.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:32.592+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:32.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:32.912+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:32.913820+0000) 2024-02-19T20:22:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:33.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:33.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:33.013+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:33.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:33.013+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:33.120+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.120899+0000) 2024-02-19T20:22:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:33.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:33.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:33.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:33.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:33.194+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:33.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:33.364+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.365451+0000) 2024-02-19T20:22:33.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:33.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:33.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:33.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:33.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:33.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:33.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:33.547+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:34.015 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:33.917+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:33.917441+0000) 2024-02-19T20:22:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:34.122+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.123042+0000) 2024-02-19T20:22:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:34.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:34.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:34.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:34.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:34.151+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:34.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:34.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:34.014+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:34.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:34.015+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:34.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:34.368+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.368419+0000) 2024-02-19T20:22:34.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:34.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:34.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:34.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:34.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:34.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:34.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:34.578+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:35.004 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:34.920+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:34.920264+0000) 2024-02-19T20:22:35.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:35.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:35.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:35.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:35.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:35.004+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:35.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:35.125+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.125813+0000) 2024-02-19T20:22:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:35.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:35.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:35.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:35.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:35.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:35.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:35.370+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.371066+0000) 2024-02-19T20:22:35.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:35.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:35.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:35.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:35.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:35.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:35.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:35.531+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:36.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:35.921+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:35.922957+0000) 2024-02-19T20:22:36.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:36.045+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:36.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:36.045+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:36.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:36.045+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:36.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:36.046+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:36.373 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:36.127+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.128399+0000) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:36.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:36.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:36.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:36.374 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:36.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:36.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:36.372+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.373113+0000) 2024-02-19T20:22:36.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:36.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:36.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:36.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:36.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:36.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:36.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:36.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:37.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:36.923+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:36.924149+0000) 2024-02-19T20:22:37.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:37.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:37.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:37.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:37.071+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:37.128+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.129829+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:37.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:37.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:37.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:37.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:37.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:37.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:37.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:37.686 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:37.566+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61390 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:37.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:37.563+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.564149+0000) 2024-02-19T20:22:37.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:37.564+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.565861+0000) 2024-02-19T20:22:37.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:37.373+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.374668+0000) 2024-02-19T20:22:37.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:37.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:37.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:37.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:37.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:37.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:37.563+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.564237+0000) 2024-02-19T20:22:37.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:37.923+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:37.924338+0000) 2024-02-19T20:22:38.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:38.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:38.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:38.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:38.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:38.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:38.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:38.083+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:38.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:38.129+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.130080+0000) 2024-02-19T20:22:38.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:38.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:38.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:38.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:38.448 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:38.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:38.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:38.197+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:38.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:38.373+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.374880+0000) 2024-02-19T20:22:38.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:38.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:38.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:38.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:38.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:38.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:38.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:38.452+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:39.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:38.925+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:38.926337+0000) 2024-02-19T20:22:39.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:39.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:39.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:39.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:39.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:39.044+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:39.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:39.045+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:39.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:39.131+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.132087+0000) 2024-02-19T20:22:39.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:39.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:39.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:39.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:39.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:39.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:39.379 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:39.205+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:39.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:39.376+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.377342+0000) 2024-02-19T20:22:39.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:39.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:39.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:39.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:39.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:39.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:39.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:39.415+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:40.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:39.928+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:39.929044+0000) 2024-02-19T20:22:40.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:40.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:40.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:40.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:40.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:40.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:40.034+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:40.133+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.134712+0000) 2024-02-19T20:22:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:40.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:40.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:40.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:40.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:40.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:40.294 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:40.162+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:40.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:40.378+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.379504+0000) 2024-02-19T20:22:40.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:40.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:40.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:40.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:40.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:40.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:40.722 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:40.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:41.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:40.930+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:40.931155+0000) 2024-02-19T20:22:41.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:41.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:41.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:41.035+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:41.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:41.035+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:41.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:41.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:41.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:41.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:41.134+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:41.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:41.136+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.137276+0000) 2024-02-19T20:22:41.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:41.382+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.383150+0000) 2024-02-19T20:22:41.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:41.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:41.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:41.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:41.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:41.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:41.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:41.474+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:42.002 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:41.931+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:41.932453+0000) 2024-02-19T20:22:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:42.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:42.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:42.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:42.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:42.293 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:42.001+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:42.138+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.139261+0000) 2024-02-19T20:22:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:42.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:42.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:42.139+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:42.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:42.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:42.693 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:42.573+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61430 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:42.694 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:42.570+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.571466+0000) 2024-02-19T20:22:42.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:42.570+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.571913+0000) 2024-02-19T20:22:42.694 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:42.383+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.384266+0000) 2024-02-19T20:22:42.694 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:42.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:42.695 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:42.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:42.695 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:42.476+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:42.695 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:42.476+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:42.695 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:42.570+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.571609+0000) 2024-02-19T20:22:42.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:42.931+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:42.932654+0000) 2024-02-19T20:22:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.030+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.030+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.030+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:43.292 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.031+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:43.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:43.138+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.139405+0000) 2024-02-19T20:22:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:43.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:43.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:43.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:43.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:43.161+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:43.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:43.383+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.384433+0000) 2024-02-19T20:22:43.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:43.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:43.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:43.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:43.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:43.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:43.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:43.482+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:43.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.933+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:43.934838+0000) 2024-02-19T20:22:43.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:43.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:43.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:44.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:43.991+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:44.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:44.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:44.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:44.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:44.135+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:44.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:44.140+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.141477+0000) 2024-02-19T20:22:44.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:44.385+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.386494+0000) 2024-02-19T20:22:44.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:44.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:44.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:44.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:44.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:44.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:44.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:44.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:45.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:44.935+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:44.937000+0000) 2024-02-19T20:22:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:44.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:44.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:44.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:44.964+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:45.388 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:45.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:45.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:45.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:45.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:45.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:45.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:45.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:45.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:45.142+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.143577+0000) 2024-02-19T20:22:45.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:45.387+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.388665+0000) 2024-02-19T20:22:45.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:45.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:45.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:45.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:45.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:45.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:45.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:45.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:46.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:45.938+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:45.939104+0000) 2024-02-19T20:22:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:45.980+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:45.980+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:45.980+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:46.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:45.981+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:46.391 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:46.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:46.391 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:46.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:46.391 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:46.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:46.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:46.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:46.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:46.145+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.146246+0000) 2024-02-19T20:22:46.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:46.390+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.391536+0000) 2024-02-19T20:22:46.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:46.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:46.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:46.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:46.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:46.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:46.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:46.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:46.939+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:46.940216+0000) 2024-02-19T20:22:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:46.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:46.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:46.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:47.044 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:46.944+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:47.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:47.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:47.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:47.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:47.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:47.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:47.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:47.068+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:47.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:47.146+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.147636+0000) 2024-02-19T20:22:47.700 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:47.581+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61469 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:47.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:47.578+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.579358+0000) 2024-02-19T20:22:47.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:47.578+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.579156+0000) 2024-02-19T20:22:47.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:47.391+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.392630+0000) 2024-02-19T20:22:47.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:47.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:47.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:47.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:47.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:47.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:47.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:47.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:47.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:47.578+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.579181+0000) 2024-02-19T20:22:47.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:47.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:47.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:47.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:47.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:47.929+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:47.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:47.930+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:47.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:47.939+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:47.940394+0000) 2024-02-19T20:22:48.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:48.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:48.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:48.035+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:48.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:48.035+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:48.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:48.146+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.147806+0000) 2024-02-19T20:22:48.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:48.391+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.392811+0000) 2024-02-19T20:22:48.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:48.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:48.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:48.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:48.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:48.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:48.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:48.463+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:49.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:48.894+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:49.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:48.894+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:49.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:48.894+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:49.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:48.894+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:49.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:48.941+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:48.942365+0000) 2024-02-19T20:22:49.394 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:49.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:49.395 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:49.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:49.395 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:49.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:49.395 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:49.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:49.395 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:49.148+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.149952+0000) 2024-02-19T20:22:49.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:49.393+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.394837+0000) 2024-02-19T20:22:49.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:49.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:49.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:49.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:49.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:50.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:49.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:50.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:49.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:49.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:49.935+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:50.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:49.943+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:49.944423+0000) 2024-02-19T20:22:50.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:50.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:50.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:50.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:50.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:50.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:50.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:50.057+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:50.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:50.151+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.152012+0000) 2024-02-19T20:22:50.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:50.396+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.397175+0000) 2024-02-19T20:22:50.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:50.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:50.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:50.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:50.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:50.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:50.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:50.423+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:51.008 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:50.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:51.008 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:50.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:51.008 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:50.916+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:51.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:50.916+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:51.009 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:50.945+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:50.946449+0000) 2024-02-19T20:22:51.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:51.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:51.293 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.153+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.154444+0000) 2024-02-19T20:22:51.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:51.398+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.399079+0000) 2024-02-19T20:22:51.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:51.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:51.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:51.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:51.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:51.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:51.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:51.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:51.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:51.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:51.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:51.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:51.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:51.907+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:51.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:51.907+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:51.990 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:51.946+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:51.947633+0000) 2024-02-19T20:22:52.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:52.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:51.989+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:52.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:52.154+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.155993+0000) 2024-02-19T20:22:52.709 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:52.588+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61509 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:52.710 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:52.585+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.586728+0000) 2024-02-19T20:22:52.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:52.585+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.586050+0000) 2024-02-19T20:22:52.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:52.400+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.401046+0000) 2024-02-19T20:22:52.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:52.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:52.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:52.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:52.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:52.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:52.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:52.445+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:52.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:52.585+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.586453+0000) 2024-02-19T20:22:52.973 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:52.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:52.973 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:52.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:52.973 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:52.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:52.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:52.866+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:52.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:52.946+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:52.947869+0000) 2024-02-19T20:22:53.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:52.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:53.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:52.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:52.972+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:52.973+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:53.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:53.155+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.156180+0000) 2024-02-19T20:22:53.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:53.400+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.401234+0000) 2024-02-19T20:22:53.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:53.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:53.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:53.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:53.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:53.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:53.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:53.447+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:53.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:53.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:54.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:53.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:53.859+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:54.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:53.948+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:53.949897+0000) 2024-02-19T20:22:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:53.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:53.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:53.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:54.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:53.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:54.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:54.157+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.158926+0000) 2024-02-19T20:22:54.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:54.402+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.403188+0000) 2024-02-19T20:22:54.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:54.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:54.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:54.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:54.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:54.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:54.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:54.430+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:54.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:54.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:55.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:54.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:55.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:54.969+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:54.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:54.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:54.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:54.873+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:55.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:54.952+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:54.953512+0000) 2024-02-19T20:22:55.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:55.161+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.162518+0000) 2024-02-19T20:22:55.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:55.404+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.405411+0000) 2024-02-19T20:22:55.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:55.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:55.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:55.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:55.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:55.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:55.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:55.420+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:55.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:55.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:56.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:55.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:56.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:55.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:56.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:55.883+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:56.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:55.955+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:55.956151+0000) 2024-02-19T20:22:56.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:55.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:56.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:55.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:56.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:55.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:55.999+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:56.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:56.163+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.164553+0000) 2024-02-19T20:22:56.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:56.406+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.407629+0000) 2024-02-19T20:22:56.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:56.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:56.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:56.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:56.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:56.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:56.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:56.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:57.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:56.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:57.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:56.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:57.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:56.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:57.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:56.861+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:57.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:56.956+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:56.957746+0000) 2024-02-19T20:22:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:57.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:57.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:57.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:57.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:57.024+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:57.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:57.165+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.166176+0000) 2024-02-19T20:22:57.717 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:22:57.596+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61548 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:22:57.717 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:57.593+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.594448+0000) 2024-02-19T20:22:57.718 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:57.593+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.594085+0000) 2024-02-19T20:22:57.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:57.408+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.409386+0000) 2024-02-19T20:22:57.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:57.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:57.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:57.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:57.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:57.437+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:57.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:57.438+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:57.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:57.593+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.594144+0000) 2024-02-19T20:22:58.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:57.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:58.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:57.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:58.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:57.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:58.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:57.818+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:58.034 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:57.956+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:57.957980+0000) 2024-02-19T20:22:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:58.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:58.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:58.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:58.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:58.033+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:58.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:58.165+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.166362+0000) 2024-02-19T20:22:58.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:58.408+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.409634+0000) 2024-02-19T20:22:58.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:58.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:58.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:58.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:58.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:58.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:58.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:58.466+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:59.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:58.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:58.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:58.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:22:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:58.856+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:22:59.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:58.959+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:58.960235+0000) 2024-02-19T20:22:59.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:59.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:59.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:59.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:59.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:59.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:22:59.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:59.068+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:22:59.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:22:59.167+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.168513+0000) 2024-02-19T20:22:59.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:59.411+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.412045+0000) 2024-02-19T20:22:59.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:59.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:59.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:59.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:59.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:59.506+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:22:59.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:22:59.507+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:00.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:59.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:00.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:59.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:00.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:59.831+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:00.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:59.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:00.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:22:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:22:59.961+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:22:59.962383+0000) 2024-02-19T20:23:00.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:00.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:00.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:00.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:00.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:00.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:00.028+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:00.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:00.169+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.170551+0000) 2024-02-19T20:23:00.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:00.414+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.415878+0000) 2024-02-19T20:23:00.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:00.523+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:00.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:00.523+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:00.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:00.523+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:00.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:00.524+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:01.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:00.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:01.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:00.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:01.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:00.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:01.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:00.837+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:01.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:00.963+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:00.964527+0000) 2024-02-19T20:23:01.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:01.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:01.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:01.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:01.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:01.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:01.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:01.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:01.171+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.172684+0000) 2024-02-19T20:23:01.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:01.416+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.417926+0000) 2024-02-19T20:23:01.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:01.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:01.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:01.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:01.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:01.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:01.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:01.557+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:02.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:01.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:01.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:01.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:01.887+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:02.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:01.964+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:01.965830+0000) 2024-02-19T20:23:02.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:02.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:02.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:02.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:02.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:02.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:02.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:02.056+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:02.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:02.173+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.174054+0000) 2024-02-19T20:23:02.723 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:02.600+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.601385+0000) 2024-02-19T20:23:02.723 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:02.419+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.420177+0000) 2024-02-19T20:23:02.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:02.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:02.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:02.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:02.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:02.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:02.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:02.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:02.724 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:02.600+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.601770+0000) 2024-02-19T20:23:02.725 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:02.604+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61588 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:02.725 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:02.601+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.602457+0000) 2024-02-19T20:23:03.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:02.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:03.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:02.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:03.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:02.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:03.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:02.914+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:03.012 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:02.965+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:02.966037+0000) 2024-02-19T20:23:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:03.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:03.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:03.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:03.012+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:03.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:03.173+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.174240+0000) 2024-02-19T20:23:03.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:03.419+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.420363+0000) 2024-02-19T20:23:03.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:03.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:03.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:03.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:03.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:03.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:03.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:03.617+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:04.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:03.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:04.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:03.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:04.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:03.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:04.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:03.892+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:04.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:03.967+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:03.968027+0000) 2024-02-19T20:23:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:04.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:04.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:04.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:04.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:04.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:04.175+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.176335+0000) 2024-02-19T20:23:04.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:04.421+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.422329+0000) 2024-02-19T20:23:04.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:04.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:04.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:04.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:04.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:04.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:04.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:04.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:05.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:04.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:05.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:04.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:05.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:04.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:05.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:04.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:05.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:04.969+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:04.970077+0000) 2024-02-19T20:23:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.023+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:05.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.177+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.178504+0000) 2024-02-19T20:23:05.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:05.423+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.424945+0000) 2024-02-19T20:23:05.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:05.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:05.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:05.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:05.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:05.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:05.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:05.630+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:05.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:05.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:05.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:05.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:05.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:05.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:05.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:05.878+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:05.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:05.971+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:05.972308+0000) 2024-02-19T20:23:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:05.999+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:06.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:06.179+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.180772+0000) 2024-02-19T20:23:06.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:06.426+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.427370+0000) 2024-02-19T20:23:06.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:06.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:06.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:06.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:06.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:06.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:06.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:06.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:06.991 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:06.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:06.991 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:06.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:06.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:06.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:06.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:06.852+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:06.992 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:06.972+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:06.973785+0000) 2024-02-19T20:23:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:06.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:06.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:06.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:06.991+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:07.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:07.180+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.181999+0000) 2024-02-19T20:23:07.729 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:07.611+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61627 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:07.729 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:07.607+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.608796+0000) 2024-02-19T20:23:07.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:07.608+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.609426+0000) 2024-02-19T20:23:07.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:07.427+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.428789+0000) 2024-02-19T20:23:07.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:07.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:07.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:07.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:07.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:07.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:07.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:07.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:07.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:07.608+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.609511+0000) 2024-02-19T20:23:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:07.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:07.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:08.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:07.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:08.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:07.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:07.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:07.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:07.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:08.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:07.899+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:08.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:07.972+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:07.973937+0000) 2024-02-19T20:23:08.428 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:08.181+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.182173+0000) 2024-02-19T20:23:08.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:08.428+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.429081+0000) 2024-02-19T20:23:08.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:08.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:08.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:08.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:08.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:08.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:08.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:08.629+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:09.003 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:08.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:09.003 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:08.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:09.003 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:08.893+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:09.003 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:08.894+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:09.003 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:08.975+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:08.976387+0000) 2024-02-19T20:23:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:09.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.183+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.184370+0000) 2024-02-19T20:23:09.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:09.430+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.431316+0000) 2024-02-19T20:23:09.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:09.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:09.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:09.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:09.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:09.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:09.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:09.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:10.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:09.967+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:10.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:09.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:09.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:09.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:09.857+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:10.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:09.977+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:09.978459+0000) 2024-02-19T20:23:10.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:10.186+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.187085+0000) 2024-02-19T20:23:10.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:10.432+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.433980+0000) 2024-02-19T20:23:10.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:10.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:10.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:10.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:10.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:10.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:10.979 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:10.714+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:10.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:10.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:10.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:10.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:10.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:10.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:10.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:10.873+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:10.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:10.979+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:10.980753+0000) 2024-02-19T20:23:11.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:10.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:10.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:10.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:10.998+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:11.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:11.189+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.190787+0000) 2024-02-19T20:23:11.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:11.436+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.437576+0000) 2024-02-19T20:23:11.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:11.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:11.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:11.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:11.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:11.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:11.995 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:11.745+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:11.995 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:11.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:11.995 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:11.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:11.996 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:11.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:11.996 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:11.897+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:11.996 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:11.981+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:11.982503+0000) 2024-02-19T20:23:11.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:11.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:11.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:11.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:11.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:11.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:11.995+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:12.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:12.191+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.192452+0000) 2024-02-19T20:23:12.736 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:12.615+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.616163+0000) 2024-02-19T20:23:12.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:12.618+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61667 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:12.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:12.615+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.616771+0000) 2024-02-19T20:23:12.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:12.437+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.438861+0000) 2024-02-19T20:23:12.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:12.615+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.616823+0000) 2024-02-19T20:23:13.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:12.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:13.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:12.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:13.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:12.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:13.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:12.739+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:13.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:12.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:13.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:12.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:13.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:12.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:13.011 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:12.903+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:13.011 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:12.981+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:12.982679+0000) 2024-02-19T20:23:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:13.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:13.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:13.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:13.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:13.010+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:13.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:13.191+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.192749+0000) 2024-02-19T20:23:13.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:13.438+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.439041+0000) 2024-02-19T20:23:13.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:13.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:13.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:13.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:13.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:13.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:13.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:13.721+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:13.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:13.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:13.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:13.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:13.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:13.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:13.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:13.926+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:13.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:13.984+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:13.985250+0000) 2024-02-19T20:23:14.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:14.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:14.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:14.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:14.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:14.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:14.193+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.194789+0000) 2024-02-19T20:23:14.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:14.440+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.441538+0000) 2024-02-19T20:23:14.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:14.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:14.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:14.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:14.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:14.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:14.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:14.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:15.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:14.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:15.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:14.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:15.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:14.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:15.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:14.966+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:15.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:14.988+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:14.989049+0000) 2024-02-19T20:23:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.001+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:15.292 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.195+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.196961+0000) 2024-02-19T20:23:15.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:15.443+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.444072+0000) 2024-02-19T20:23:15.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:15.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:15.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:15.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:15.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:15.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:15.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:15.944+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:15.989 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:15.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:15.989 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:15.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:15.990 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:15.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:15.990 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:15.722+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:16.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:15.991+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:15.992743+0000) 2024-02-19T20:23:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:15.989+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:16.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:16.198+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.199160+0000) 2024-02-19T20:23:16.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:16.445+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.446121+0000) 2024-02-19T20:23:16.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:16.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:16.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:16.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:16.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:16.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:16.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:16.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:16.994 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:16.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:16.994 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:16.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:16.994 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:16.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:16.994 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:16.956+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:17.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:17.199+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.200408+0000) 2024-02-19T20:23:17.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:16.993+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:16.994103+0000) 2024-02-19T20:23:17.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:16.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:17.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:16.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:17.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:16.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:17.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:16.993+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:17.736 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:17.626+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61706 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:17.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:17.622+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.623247+0000) 2024-02-19T20:23:17.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:17.623+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.624587+0000) 2024-02-19T20:23:17.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:17.446+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.447280+0000) 2024-02-19T20:23:17.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:17.623+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.624340+0000) 2024-02-19T20:23:17.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:17.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:17.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:17.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:17.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:17.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:17.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:17.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:17.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:17.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:17.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:17.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:17.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:17.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:17.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:17.975+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:17.994 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:17.993+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:17.994200+0000) 2024-02-19T20:23:17.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:17.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:17.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:17.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:17.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:17.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:17.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:17.983+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:18.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:18.199+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.200580+0000) 2024-02-19T20:23:18.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:18.446+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.447431+0000) 2024-02-19T20:23:18.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:18.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:18.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:18.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:18.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:18.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:18.987 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:18.727+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:18.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:18.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:18.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:18.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:18.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:18.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:18.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:18.951+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:19.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:19.202+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.203108+0000) 2024-02-19T20:23:19.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:18.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:19.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:18.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:19.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:18.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:19.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:18.987+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:19.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:18.996+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:18.997809+0000) 2024-02-19T20:23:19.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:19.448+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:19.449786+0000) 2024-02-19T20:23:20.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:19.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:20.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:19.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:20.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:19.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:20.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:19.979+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:20.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:19.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:20.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:19.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:20.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:19.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:20.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:19.776+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:20.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:19.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:20.002 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:19.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:20.002 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:19.916+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:20.002 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:19.917+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:20.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:20.000+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.001248+0000) 2024-02-19T20:23:20.291 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:20.205+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.205138+0000) 2024-02-19T20:23:20.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:20.453+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:20.453153+0000) 2024-02-19T20:23:21.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:20.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:21.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:20.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:21.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:20.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:21.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:20.813+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:21.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:20.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:21.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:20.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:21.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:20.938+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:21.005 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:20.938+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:21.005 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:20.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:21.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:20.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:21.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:20.901+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:21.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:20.902+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:21.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:21.207+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.207365+0000) 2024-02-19T20:23:21.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:21.005+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.005044+0000) 2024-02-19T20:23:21.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:21.455+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:21.455186+0000) 2024-02-19T20:23:22.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:21.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:22.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:21.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:22.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:21.836+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:22.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:21.837+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:22.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:21.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:22.006 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:21.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:22.007 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:21.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:22.007 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:21.888+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:22.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:21.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:22.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:21.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:22.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:21.920+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:22.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:21.920+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:22.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:22.207+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.208558+0000) 2024-02-19T20:23:22.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:22.005+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.006838+0000) 2024-02-19T20:23:22.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:22.633+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61746 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:22.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:22.630+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.631422+0000) 2024-02-19T20:23:22.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:22.630+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.631397+0000) 2024-02-19T20:23:22.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:22.455+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.456810+0000) 2024-02-19T20:23:22.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:22.629+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:22.630870+0000) 2024-02-19T20:23:23.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:22.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:23.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:22.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:23.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:22.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:23.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:22.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:23.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:22.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:23.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:22.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:23.007 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:22.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:23.008 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:22.933+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:23.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:22.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:23.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:22.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:23.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:22.858+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:23.008 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:22.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:23.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:23.207+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.208777+0000) 2024-02-19T20:23:23.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:23.006+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.007060+0000) 2024-02-19T20:23:23.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:23.456+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:23.457018+0000) 2024-02-19T20:23:24.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:23.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:24.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:23.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:24.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:23.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:24.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:23.787+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:24.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:23.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:24.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:23.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:24.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:23.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:24.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:23.926+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:24.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:23.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:24.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:23.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:24.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:23.867+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:24.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:23.868+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:24.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:24.211+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.212148+0000) 2024-02-19T20:23:24.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:24.009+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.010509+0000) 2024-02-19T20:23:24.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:24.457+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:24.458973+0000) 2024-02-19T20:23:25.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:24.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:25.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:24.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:25.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:24.785+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:25.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:24.785+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:25.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:24.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:25.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:24.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:25.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:24.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:25.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:24.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:25.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:24.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:25.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:24.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:25.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:24.925+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:25.014 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:24.926+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:25.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:25.213+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.214247+0000) 2024-02-19T20:23:25.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:25.013+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.014083+0000) 2024-02-19T20:23:25.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:25.461+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:25.462473+0000) 2024-02-19T20:23:26.015 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:25.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:26.015 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:25.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:26.015 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:25.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:26.015 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:25.937+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:26.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:25.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:26.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:25.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:26.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:25.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:26.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:25.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:26.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:25.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:26.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:25.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:26.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:25.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:26.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:25.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:26.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:26.215+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.216300+0000) 2024-02-19T20:23:26.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:26.015+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.016235+0000) 2024-02-19T20:23:26.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:26.464+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:26.465918+0000) 2024-02-19T20:23:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:26.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:26.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:26.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:27.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:26.788+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:27.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:26.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:27.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:26.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:27.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:26.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:27.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:26.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:27.018 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:26.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:27.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:26.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:27.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:26.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:27.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:26.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:27.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:27.216+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.217690+0000) 2024-02-19T20:23:27.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:27.017+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.018449+0000) 2024-02-19T20:23:27.737 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:27.640+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61785 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:27.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:27.637+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.638239+0000) 2024-02-19T20:23:27.738 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:27.636+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.637882+0000) 2024-02-19T20:23:27.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:27.467+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.468064+0000) 2024-02-19T20:23:27.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:27.637+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:27.638617+0000) 2024-02-19T20:23:28.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:27.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:28.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:27.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:28.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:27.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:28.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:27.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:28.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:27.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:28.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:27.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:28.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:27.865+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:28.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:27.866+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:28.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:27.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:28.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:27.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:28.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:27.943+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:28.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:27.943+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:28.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:28.216+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.217978+0000) 2024-02-19T20:23:28.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:28.017+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.018667+0000) 2024-02-19T20:23:28.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:28.467+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:28.468247+0000) 2024-02-19T20:23:29.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:28.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:29.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:28.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:29.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:28.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:29.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:28.898+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:29.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:28.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:29.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:28.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:29.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:28.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:29.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:28.913+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:29.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:28.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:29.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:28.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:29.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:28.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:29.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:28.792+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:29.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:29.219+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.220201+0000) 2024-02-19T20:23:29.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:29.021+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.022262+0000) 2024-02-19T20:23:29.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:29.469+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:29.470917+0000) 2024-02-19T20:23:30.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:29.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:30.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:29.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:30.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:29.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:30.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:29.930+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:30.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:29.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:30.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:29.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:30.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:29.756+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:30.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:29.757+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:30.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:29.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:30.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:29.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:30.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:29.875+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:30.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:29.876+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:30.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:30.221+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.222433+0000) 2024-02-19T20:23:30.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:30.023+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.024935+0000) 2024-02-19T20:23:30.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:30.471+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:30.472887+0000) 2024-02-19T20:23:31.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:30.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:31.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:30.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:31.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:30.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:31.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:30.741+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:31.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:30.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:31.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:30.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:31.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:30.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:31.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:30.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:31.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:30.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:31.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:30.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:31.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:30.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:31.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:30.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:31.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:31.223+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.224463+0000) 2024-02-19T20:23:31.291 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:31.026+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.027270+0000) 2024-02-19T20:23:31.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:31.474+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:31.475340+0000) 2024-02-19T20:23:32.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:31.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:32.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:31.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:32.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:31.787+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:32.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:31.787+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:32.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:31.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:32.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:31.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:32.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:31.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:32.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:31.882+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:32.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:31.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:32.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:31.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:32.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:31.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:32.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:31.936+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:32.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:32.225+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.226043+0000) 2024-02-19T20:23:32.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:32.027+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.028747+0000) 2024-02-19T20:23:32.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:32.647+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61824 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:32.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:32.646+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.647191+0000) 2024-02-19T20:23:32.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:32.644+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.645787+0000) 2024-02-19T20:23:32.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:32.475+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.476849+0000) 2024-02-19T20:23:32.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:32.644+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:32.645251+0000) 2024-02-19T20:23:33.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:32.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:33.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:32.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:33.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:32.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:33.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:32.802+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:33.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:32.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:33.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:32.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:33.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:32.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:33.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:32.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:33.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:32.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:33.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:32.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:33.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:32.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:33.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:32.866+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:33.030 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:33.027+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.028965+0000) 2024-02-19T20:23:33.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:33.225+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.226353+0000) 2024-02-19T20:23:33.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:33.476+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:33.477060+0000) 2024-02-19T20:23:34.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:33.807+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:34.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:33.807+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:34.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:33.807+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:34.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:33.807+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:34.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:33.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:34.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:33.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:34.030 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:33.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:34.031 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:33.928+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:34.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:33.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:34.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:33.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:34.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:33.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:34.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:33.884+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:34.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:34.030+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.031040+0000) 2024-02-19T20:23:34.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:34.227+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.228353+0000) 2024-02-19T20:23:34.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:34.479+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:34.480503+0000) 2024-02-19T20:23:35.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:34.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:35.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:34.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:35.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:34.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:35.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:34.823+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:35.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:34.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:35.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:34.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:35.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:34.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:35.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:34.836+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:35.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:34.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:35.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:34.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:35.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:34.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:35.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:34.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:35.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:35.229+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.230431+0000) 2024-02-19T20:23:35.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:35.032+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.033413+0000) 2024-02-19T20:23:35.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:35.481+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:35.482660+0000) 2024-02-19T20:23:36.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:35.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:36.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:35.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:36.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:35.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:36.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:35.871+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:36.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:35.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:36.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:35.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:36.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:35.804+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:36.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:35.804+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:36.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:35.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:36.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:35.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:36.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:35.894+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:36.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:35.894+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:36.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:36.231+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.232681+0000) 2024-02-19T20:23:36.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:36.036+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.037039+0000) 2024-02-19T20:23:36.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:36.483+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:36.484790+0000) 2024-02-19T20:23:37.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:36.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:37.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:36.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:37.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:36.838+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:37.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:36.838+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:37.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:36.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:37.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:36.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:37.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:36.940+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:37.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:36.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:37.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:36.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:37.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:36.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:37.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:36.762+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:37.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:36.762+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:37.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:37.233+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.234012+0000) 2024-02-19T20:23:37.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:37.037+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.038644+0000) 2024-02-19T20:23:37.738 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:37.655+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61864 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:37.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:37.652+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.653572+0000) 2024-02-19T20:23:37.739 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:37.652+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.653470+0000) 2024-02-19T20:23:37.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:37.485+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.486087+0000) 2024-02-19T20:23:37.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:37.652+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:37.653044+0000) 2024-02-19T20:23:38.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:37.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:38.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:37.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:38.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:37.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:38.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:37.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:38.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:37.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:38.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:37.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:38.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:37.801+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:38.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:37.802+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:38.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:37.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:38.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:37.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:38.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:37.888+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:38.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:37.889+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:38.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:38.037+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.038855+0000) 2024-02-19T20:23:38.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:38.233+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.234198+0000) 2024-02-19T20:23:38.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:38.485+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:38.486277+0000) 2024-02-19T20:23:39.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:38.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:39.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:38.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:39.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:38.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:39.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:38.822+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:39.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:38.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:39.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:38.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:39.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:38.890+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:39.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:38.891+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:39.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:38.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:38.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:38.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:39.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:38.884+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:39.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:39.235+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.236226+0000) 2024-02-19T20:23:39.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:39.041+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.042248+0000) 2024-02-19T20:23:39.738 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:39.487+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:39.488299+0000) 2024-02-19T20:23:40.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:39.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:40.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:39.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:40.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:39.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:40.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:39.773+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:40.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:39.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:40.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:39.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:40.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:39.871+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:40.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:39.872+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:40.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:39.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:39.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:39.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:40.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:39.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:40.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:40.237+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.238239+0000) 2024-02-19T20:23:40.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:40.044+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.045888+0000) 2024-02-19T20:23:40.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:40.490+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:40.491723+0000) 2024-02-19T20:23:41.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:40.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:41.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:40.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:41.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:40.872+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:41.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:40.872+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:41.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:40.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:41.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:40.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:41.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:40.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:41.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:40.761+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:40.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:40.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:40.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:41.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:40.873+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:41.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:41.239+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.240666+0000) 2024-02-19T20:23:41.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:41.046+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.047907+0000) 2024-02-19T20:23:41.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:41.493+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:41.494340+0000) 2024-02-19T20:23:42.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:41.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:42.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:41.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:42.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:41.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:42.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:41.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:42.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:41.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:42.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:41.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:42.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:41.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:42.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:41.865+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:42.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:41.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:41.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:41.764+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:42.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:41.765+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:42.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:42.240+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.241945+0000) 2024-02-19T20:23:42.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:42.049+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.050133+0000) 2024-02-19T20:23:42.749 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:42.662+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61903 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:42.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:42.494+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.495919+0000) 2024-02-19T20:23:42.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:42.660+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.661728+0000) 2024-02-19T20:23:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:42.659+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.660776+0000) 2024-02-19T20:23:42.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:42.659+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:42.660291+0000) 2024-02-19T20:23:42.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:42.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:42.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:42.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:42.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:42.750+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:43.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:42.750+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:43.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:42.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:43.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:42.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:42.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:43.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:42.843+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:43.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:42.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:43.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:42.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:43.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:42.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:43.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:42.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:43.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:43.049+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.050314+0000) 2024-02-19T20:23:43.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:43.241+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.242132+0000) 2024-02-19T20:23:43.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:43.495+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:43.496138+0000) 2024-02-19T20:23:43.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:43.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:43.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:43.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:43.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:43.741+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:43.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:43.741+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:44.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:43.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:44.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:43.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:44.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:43.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:44.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:43.830+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:44.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:43.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:44.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:43.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:44.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:43.832+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:44.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:43.833+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:44.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:44.051+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.052883+0000) 2024-02-19T20:23:44.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:44.243+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.244253+0000) 2024-02-19T20:23:44.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:44.497+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:44.498721+0000) 2024-02-19T20:23:45.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:44.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:45.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:44.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:45.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:44.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:45.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:44.824+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:45.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:44.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:45.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:44.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:45.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:44.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:45.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:44.876+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:45.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:44.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:44.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:44.765+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:45.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:44.765+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:45.315 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:45.245+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.246920+0000) 2024-02-19T20:23:45.315 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:45.053+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.054956+0000) 2024-02-19T20:23:45.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:45.501+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:45.502248+0000) 2024-02-19T20:23:46.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:45.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:46.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:45.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:46.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:45.832+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:46.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:45.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:46.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:45.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:46.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:45.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:46.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:45.756+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:46.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:45.756+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:45.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:46.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:45.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:45.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:46.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:45.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:46.504 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:46.056+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.057037+0000) 2024-02-19T20:23:46.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:46.248+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.249103+0000) 2024-02-19T20:23:46.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:46.504+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:46.505932+0000) 2024-02-19T20:23:47.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:46.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:47.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:46.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:47.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:46.861+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:47.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:46.862+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:46.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:46.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:46.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:47.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:46.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:47.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:46.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:47.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:46.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:47.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:46.787+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:47.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:46.788+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:47.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:47.249+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.250369+0000) 2024-02-19T20:23:47.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:47.057+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.058763+0000) 2024-02-19T20:23:47.758 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:47.670+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61943 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:47.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:47.666+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.667981+0000) 2024-02-19T20:23:47.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:47.666+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.667350+0000) 2024-02-19T20:23:47.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:47.507+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.508144+0000) 2024-02-19T20:23:47.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:47.666+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:47.668007+0000) 2024-02-19T20:23:48.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:47.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:47.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:47.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:48.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:47.817+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:47.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:47.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:48.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:47.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:48.043 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:47.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:47.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:47.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:48.043 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:47.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:48.044 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:47.824+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:48.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:48.057+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.058966+0000) 2024-02-19T20:23:48.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:48.249+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.250529+0000) 2024-02-19T20:23:48.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:48.507+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:48.508315+0000) 2024-02-19T20:23:49.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:48.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:49.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:48.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:49.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:48.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:49.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:48.793+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:49.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:48.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:49.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:48.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:49.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:48.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:49.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:48.858+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:49.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:48.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:49.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:48.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:48.801+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:49.042 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:48.801+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:49.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:49.251+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.252527+0000) 2024-02-19T20:23:49.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:49.060+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.061141+0000) 2024-02-19T20:23:49.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:49.509+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:49.510535+0000) 2024-02-19T20:23:50.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:49.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:50.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:49.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:50.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:49.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:50.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:49.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:50.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:49.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:50.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:49.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:50.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:49.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:50.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:49.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:50.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:49.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:50.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:49.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:49.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:50.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:49.836+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:50.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:50.254+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.255155+0000) 2024-02-19T20:23:50.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:50.062+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.063702+0000) 2024-02-19T20:23:50.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:50.512+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:50.513188+0000) 2024-02-19T20:23:50.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:50.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:50.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:50.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:50.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:50.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:51.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:50.766+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:51.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:50.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:51.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:50.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:51.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:50.814+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:51.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:50.815+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:50.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:50.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:50.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:51.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:50.861+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:51.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:51.256+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.257100+0000) 2024-02-19T20:23:51.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:51.064+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.065985+0000) 2024-02-19T20:23:51.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:51.514+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:51.515691+0000) 2024-02-19T20:23:51.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:51.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:51.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:51.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:52.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:51.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:52.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:51.727+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:52.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:51.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:52.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:51.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:52.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:51.770+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:52.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:51.771+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:51.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:51.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:51.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:52.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:51.892+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:52.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:52.066+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.067726+0000) 2024-02-19T20:23:52.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:52.257+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.258334+0000) 2024-02-19T20:23:52.789 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:52.677+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 61982 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:52.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:52.674+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.675195+0000) 2024-02-19T20:23:52.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:52.674+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.675141+0000) 2024-02-19T20:23:52.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:52.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:52.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:52.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:52.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:52.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:52.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:52.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:52.516+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.517935+0000) 2024-02-19T20:23:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:52.674+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:52.675422+0000) 2024-02-19T20:23:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:52.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:52.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:52.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:52.792 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:52.708+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:53.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:52.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:53.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:52.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:53.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:52.918+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:53.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:52.919+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:53.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:53.257+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.258494+0000) 2024-02-19T20:23:53.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:53.066+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.067905+0000) 2024-02-19T20:23:53.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:53.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:53.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:53.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:53.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:53.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:53.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:53.767+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:53.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:53.517+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:53.518150+0000) 2024-02-19T20:23:53.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:53.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:53.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:53.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:53.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:53.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:53.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:53.697+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:54.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:53.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:54.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:53.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:54.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:53.912+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:54.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:53.912+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:54.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:54.259+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.260616+0000) 2024-02-19T20:23:54.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:54.069+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.070348+0000) 2024-02-19T20:23:54.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:54.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:54.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:54.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:54.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:54.754+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:54.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:54.754+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:54.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:54.520+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:54.521717+0000) 2024-02-19T20:23:54.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:54.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:54.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:54.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:54.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:54.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:54.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:54.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:55.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:54.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:55.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:54.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:55.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:54.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:55.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:54.948+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:55.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:55.261+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.262672+0000) 2024-02-19T20:23:55.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:55.071+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.072384+0000) 2024-02-19T20:23:55.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:55.523+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:55.524267+0000) 2024-02-19T20:23:55.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:55.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:55.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:55.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:55.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:55.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:55.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:55.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:56.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:55.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:56.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:55.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:56.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:55.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:56.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:55.997+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:56.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:55.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:56.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:55.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:56.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:55.773+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:56.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:55.774+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:56.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:56.073+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.074390+0000) 2024-02-19T20:23:56.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:56.265+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.266340+0000) 2024-02-19T20:23:56.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:56.525+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:56.526755+0000) 2024-02-19T20:23:56.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:56.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:56.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:56.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:56.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:56.739+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:56.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:56.740+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:57.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:56.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:57.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:56.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:57.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:56.776+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:57.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:56.777+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:57.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:57.266+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.267653+0000) 2024-02-19T20:23:57.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:57.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:57.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:57.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:57.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:57.047+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:57.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:57.047+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:57.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:57.074+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.075831+0000) 2024-02-19T20:23:57.786 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:57.681+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.682421+0000) 2024-02-19T20:23:57.787 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:57.681+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.682191+0000) 2024-02-19T20:23:57.787 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:23:57.685+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62022 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:23:57.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:57.527+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.528213+0000) 2024-02-19T20:23:57.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:57.681+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:57.682692+0000) 2024-02-19T20:23:57.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:57.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:57.788 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:57.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:57.788 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:57.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:57.788 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:57.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:58.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:58.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:58.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:58.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:58.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:58.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:58.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:58.020+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:58.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:57.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:57.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:57.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:58.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:57.788+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:58.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:58.266+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.267840+0000) 2024-02-19T20:23:58.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:58.075+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.076011+0000) 2024-02-19T20:23:58.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:58.527+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:58.528395+0000) 2024-02-19T20:23:58.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:58.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:58.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:58.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:58.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:58.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:58.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:58.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:59.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:59.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:59.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:59.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:59.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:59.004+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:23:59.063 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:59.004+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:23:59.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:58.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:59.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:58.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:59.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:58.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:23:59.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:58.832+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:23:59.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:23:59.077+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.078183+0000) 2024-02-19T20:23:59.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:59.268+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.269892+0000) 2024-02-19T20:23:59.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:59.529+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:23:59.530627+0000) 2024-02-19T20:23:59.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:59.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:59.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:59.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:59.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:59.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:23:59.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:23:59.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:00.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:59.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:00.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:59.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:00.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:59.817+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:00.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:23:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:23:59.818+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:00.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:00.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:00.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:00.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:00.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:00.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:00.272 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:00.019+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:00.272 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:00.080+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.081611+0000) 2024-02-19T20:24:00.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:00.272+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.273502+0000) 2024-02-19T20:24:00.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:00.531+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:00.532633+0000) 2024-02-19T20:24:00.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:00.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:00.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:00.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:00.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:00.755+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:00.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:00.756+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:01.063 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:00.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:01.063 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:00.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:01.063 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:00.797+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:01.063 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:00.798+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:01.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:01.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:01.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:01.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:01.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:01.005+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:01.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:01.005+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:01.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:01.082+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.083720+0000) 2024-02-19T20:24:01.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:01.274+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.275589+0000) 2024-02-19T20:24:01.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:01.533+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:01.534690+0000) 2024-02-19T20:24:02.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:01.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:02.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:01.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:02.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:01.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:02.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:01.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:02.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:02.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:02.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:02.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.018+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:02.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:01.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:02.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:01.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:02.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:01.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:02.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:01.796+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:02.533 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.084+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.085064+0000) 2024-02-19T20:24:02.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:02.275+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.276914+0000) 2024-02-19T20:24:02.789 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:02.693+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62061 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:02.790 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.690+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.691145+0000) 2024-02-19T20:24:02.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:02.535+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.536058+0000) 2024-02-19T20:24:02.790 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:02.689+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.690897+0000) 2024-02-19T20:24:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:02.689+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:02.690760+0000) 2024-02-19T20:24:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:02.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:02.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:02.766+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:02.791 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:02.767+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:03.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:02.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:03.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:02.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:03.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:02.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:03.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:02.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:03.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:03.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:03.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:03.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:02.980+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:03.534 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:03.276+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.277115+0000) 2024-02-19T20:24:03.534 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:03.084+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.085237+0000) 2024-02-19T20:24:03.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:03.535+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:03.536271+0000) 2024-02-19T20:24:03.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:03.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:03.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:03.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:03.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:03.735+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:03.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:03.736+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:04.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:03.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:04.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:03.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:04.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:03.820+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:04.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:03.821+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:04.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:03.986+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:04.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:03.986+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:04.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:03.986+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:04.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:03.986+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:04.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:04.279+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.280577+0000) 2024-02-19T20:24:04.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:04.086+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.087264+0000) 2024-02-19T20:24:04.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:04.537+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:04.538454+0000) 2024-02-19T20:24:04.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:04.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:04.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:04.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:04.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:04.728+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:04.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:04.729+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:05.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:04.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:05.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:04.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:05.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:04.812+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:05.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:04.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:05.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:05.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:05.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:05.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:05.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:05.018+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:05.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:05.019+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:05.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:05.088+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.089909+0000) 2024-02-19T20:24:05.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:05.283+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.284050+0000) 2024-02-19T20:24:05.789 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:05.539+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:05.540536+0000) 2024-02-19T20:24:05.789 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:05.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:05.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:05.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:05.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:05.758+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:05.790 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:05.759+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:06.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:05.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:06.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:05.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:06.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:05.848+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:06.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:05.849+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:06.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:06.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:06.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:06.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:06.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:06.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:06.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:06.001+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:06.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:06.285+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.286108+0000) 2024-02-19T20:24:06.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:06.091+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.092439+0000) 2024-02-19T20:24:06.791 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:06.541+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:06.542839+0000) 2024-02-19T20:24:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:06.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:06.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:06.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:06.792 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:06.764+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:07.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:06.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:07.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:06.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:07.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:06.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:07.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:06.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:07.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:07.017+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:07.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:07.017+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:07.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:07.017+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:07.065 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:07.017+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:07.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:07.287+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.287457+0000) 2024-02-19T20:24:07.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:07.094+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.094191+0000) 2024-02-19T20:24:07.803 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:07.702+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62101 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:07.804 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:07.698+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.698507+0000) 2024-02-19T20:24:07.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:07.544+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.544345+0000) 2024-02-19T20:24:07.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:07.700+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.700213+0000) 2024-02-19T20:24:07.804 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:07.699+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:07.699059+0000) 2024-02-19T20:24:07.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:07.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:07.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:07.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:07.805 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:07.806+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:08.063 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:07.807+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:08.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:07.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:08.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:07.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:08.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:07.827+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:08.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:07.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:08.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:08.286+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.287733+0000) 2024-02-19T20:24:08.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:08.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:08.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:08.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:08.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:08.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:08.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:08.066+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:08.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:08.093+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.094420+0000) 2024-02-19T20:24:08.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:08.543+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:08.544517+0000) 2024-02-19T20:24:09.063 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:08.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:09.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:08.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:09.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:08.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:09.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:08.814+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:09.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:08.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:09.064 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:08.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:09.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:08.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:09.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:08.854+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:09.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:09.288+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.289825+0000) 2024-02-19T20:24:09.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:09.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:09.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:09.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:09.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:09.073+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:09.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:09.073+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:09.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:09.095+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.096472+0000) 2024-02-19T20:24:09.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:09.546+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:09.547941+0000) 2024-02-19T20:24:10.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:09.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:10.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:09.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:10.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:09.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:10.040 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:09.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:10.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:09.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:10.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:09.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:10.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:09.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:10.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:09.830+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:10.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:10.290+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.291875+0000) 2024-02-19T20:24:10.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:10.098+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.099913+0000) 2024-02-19T20:24:10.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:10.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:10.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:10.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:10.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:10.111+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:10.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:10.111+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:10.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:10.550+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:10.551414+0000) 2024-02-19T20:24:10.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:10.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:10.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:10.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:10.871 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:10.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:10.871 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:10.787+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:11.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:11.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:11.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:11.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:11.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:11.070+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:11.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:11.071+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:11.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:11.100+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.101978+0000) 2024-02-19T20:24:11.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:10.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:11.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:10.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:11.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:10.871+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:11.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:10.872+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:11.552 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:11.292+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.293993+0000) 2024-02-19T20:24:11.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:11.553+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:11.554790+0000) 2024-02-19T20:24:11.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:11.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:11.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:11.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:12.100 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:11.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:12.100 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:11.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:12.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:11.862+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:12.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:11.862+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:12.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:11.862+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:12.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:11.862+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:12.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:12.294+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.295650+0000) 2024-02-19T20:24:12.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:12.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:12.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:12.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:12.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:12.101+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:12.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:12.102+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:12.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:12.103+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.104345+0000) 2024-02-19T20:24:12.824 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:12.708+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62140 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:12.825 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:12.705+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.706174+0000) 2024-02-19T20:24:12.825 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:12.705+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.706766+0000) 2024-02-19T20:24:12.825 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:12.555+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.556991+0000) 2024-02-19T20:24:12.825 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:12.705+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:12.706778+0000) 2024-02-19T20:24:12.825 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:12.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:12.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:12.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:12.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:12.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:12.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:12.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:13.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:12.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:13.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:12.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:13.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:12.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:13.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:12.843+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:13.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:13.294+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.295851+0000) 2024-02-19T20:24:13.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:13.077+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:13.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:13.077+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:13.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:13.077+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:13.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:13.078+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:13.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:13.103+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.104518+0000) 2024-02-19T20:24:13.817 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:13.556+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:13.557119+0000) 2024-02-19T20:24:13.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:13.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:13.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:13.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:13.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:13.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:13.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:13.767+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:14.088 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:13.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:14.088 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:13.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:14.088 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:13.819+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:14.088 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:13.820+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:14.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:14.296+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.297834+0000) 2024-02-19T20:24:14.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:14.089+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:14.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:14.089+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:14.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:14.089+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:14.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:14.090+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:14.540 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:14.106+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.107144+0000) 2024-02-19T20:24:14.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:14.558+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:14.559641+0000) 2024-02-19T20:24:14.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:14.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:14.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:14.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:14.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:14.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:14.857 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:14.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:15.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:15.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:15.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:15.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:15.109 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:15.079+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:15.109 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:15.079+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:15.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:14.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:15.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:14.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:15.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:14.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:15.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:14.858+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:15.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:15.298+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.299896+0000) 2024-02-19T20:24:15.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:15.109+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.110763+0000) 2024-02-19T20:24:15.886 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:15.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:15.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:15.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:15.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:15.888+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:15.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:15.560+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:15.561785+0000) 2024-02-19T20:24:15.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:15.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:15.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:15.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:15.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:15.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:15.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:15.741+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:16.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:15.889+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:16.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:16.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:16.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:16.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:16.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:16.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:16.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:16.035+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:16.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:16.112+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.113540+0000) 2024-02-19T20:24:16.561 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:16.300+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.301959+0000) 2024-02-19T20:24:16.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:16.563+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:16.564452+0000) 2024-02-19T20:24:16.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:16.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:16.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:16.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:16.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:16.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:16.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:16.755+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:17.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:16.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:17.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:16.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:17.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:16.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:17.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:16.890+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:17.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:17.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:17.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.028+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:17.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.029+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:17.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.114+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.115090+0000) 2024-02-19T20:24:17.563 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:17.302+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.303269+0000) 2024-02-19T20:24:17.831 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.712+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.713531+0000) 2024-02-19T20:24:17.831 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:17.712+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.713633+0000) 2024-02-19T20:24:17.832 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:17.715+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62180 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:17.832 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:17.564+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.565827+0000) 2024-02-19T20:24:17.832 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:17.713+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:17.714136+0000) 2024-02-19T20:24:17.833 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:17.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:17.833 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:17.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:17.833 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:17.723+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:17.833 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:17.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:18.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:18.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:18.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.997+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:18.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:17.998+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:18.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:17.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:18.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:17.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:18.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:17.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:18.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:17.929+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:18.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:18.302+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.303457+0000) 2024-02-19T20:24:18.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:18.114+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.115291+0000) 2024-02-19T20:24:18.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:18.565+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:18.566058+0000) 2024-02-19T20:24:18.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:18.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:18.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:18.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:18.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:18.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:18.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:18.745+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:19.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:18.981+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:19.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:18.981+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:19.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:18.981+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:19.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:18.981+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:19.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:19.116+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.117592+0000) 2024-02-19T20:24:19.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:18.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:19.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:18.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:19.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:18.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:19.290 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:18.900+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:19.566 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:19.305+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.307007+0000) 2024-02-19T20:24:19.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:19.568+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:19.569473+0000) 2024-02-19T20:24:19.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:19.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:19.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:19.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:19.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:19.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:19.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:19.776+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:20.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:19.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:20.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:19.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:20.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:19.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:20.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:19.991+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:20.235 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:20.118+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.119666+0000) 2024-02-19T20:24:20.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:19.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:20.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:19.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:20.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:19.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:20.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:19.923+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:20.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:20.308+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.309010+0000) 2024-02-19T20:24:20.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:20.571+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:20.572801+0000) 2024-02-19T20:24:20.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:20.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:20.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:20.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:20.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:20.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:20.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:20.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:21.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:20.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:21.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:20.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:21.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:20.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:21.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:20.911+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:21.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:20.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:21.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:20.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:21.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:20.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:21.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:20.944+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:21.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:21.120+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.121730+0000) 2024-02-19T20:24:21.572 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:21.310+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.311676+0000) 2024-02-19T20:24:21.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:21.573+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:21.574830+0000) 2024-02-19T20:24:21.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:21.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:21.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:21.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:21.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:21.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:21.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:21.751+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:21.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:21.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:21.936+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:22.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:21.936+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:21.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:21.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:21.964+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:21.965+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:22.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:22.122+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.123118+0000) 2024-02-19T20:24:22.573 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:22.312+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.313150+0000) 2024-02-19T20:24:22.838 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:22.723+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62219 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:22.838 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:22.719+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.720643+0000) 2024-02-19T20:24:22.839 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:22.720+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.721664+0000) 2024-02-19T20:24:22.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:22.575+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.576275+0000) 2024-02-19T20:24:22.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:22.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:22.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:22.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:22.839 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:22.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:22.840 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:22.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:22.840 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:22.719+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:22.720617+0000) 2024-02-19T20:24:23.120 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:22.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:23.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:22.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:23.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:22.917+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:23.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:22.918+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:23.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:22.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:23.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:22.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:23.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:22.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:23.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:22.910+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:23.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:23.312+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.313328+0000) 2024-02-19T20:24:23.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:23.122+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.123291+0000) 2024-02-19T20:24:23.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:23.575+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:23.576442+0000) 2024-02-19T20:24:23.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:23.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:23.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:23.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:23.895 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:23.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:23.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:23.679+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:24.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:23.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:24.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:23.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:24.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:23.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:24.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:23.897+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:24.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:23.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:24.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:23.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:24.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:23.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:24.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:23.937+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:24.290 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:24.125+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.126952+0000) 2024-02-19T20:24:24.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:24.314+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.315389+0000) 2024-02-19T20:24:24.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:24.578+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:24.579040+0000) 2024-02-19T20:24:24.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:24.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:24.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:24.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:24.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:24.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:24.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:24.711+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:25.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:24.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:25.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:24.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:25.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:24.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:25.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:24.861+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:25.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:24.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:25.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:24.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:25.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:24.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:25.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:24.950+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:25.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:25.316+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.317848+0000) 2024-02-19T20:24:25.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:25.128+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.129054+0000) 2024-02-19T20:24:25.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:25.580+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:25.581515+0000) 2024-02-19T20:24:25.843 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:25.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:25.844 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:25.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:25.844 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:25.742+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:25.844 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:25.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:26.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:25.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:26.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:25.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:26.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:25.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:26.129 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:25.952+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:26.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:25.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:26.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:25.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:26.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:25.845+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:26.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:25.846+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:26.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:26.319+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.320464+0000) 2024-02-19T20:24:26.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:26.130+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.131207+0000) 2024-02-19T20:24:26.871 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:26.582+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:26.583940+0000) 2024-02-19T20:24:26.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:26.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:26.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:26.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:26.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:26.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:26.872 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:26.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:27.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:26.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:27.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:26.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:27.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:26.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:27.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:26.966+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:27.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:26.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:27.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:26.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:27.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:26.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:27.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:26.874+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:27.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:27.321+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.322064+0000) 2024-02-19T20:24:27.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:27.132+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.133630+0000) 2024-02-19T20:24:27.846 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:27.727+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.728200+0000) 2024-02-19T20:24:27.847 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:27.730+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62259 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:27.847 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:27.726+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.727977+0000) 2024-02-19T20:24:27.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:27.584+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.585740+0000) 2024-02-19T20:24:27.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:27.726+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:27.727899+0000) 2024-02-19T20:24:27.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:27.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:27.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:27.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:27.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:27.736+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:27.848 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:27.737+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:28.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:27.946+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:28.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:27.946+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:28.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:27.946+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:28.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:27.946+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:28.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:27.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:28.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:27.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:28.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:27.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:28.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:27.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:28.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:28.132+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.133828+0000) 2024-02-19T20:24:28.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:28.321+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.322257+0000) 2024-02-19T20:24:28.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:28.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:28.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:28.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:28.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:28.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:28.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:28.584+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:28.585951+0000) 2024-02-19T20:24:28.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:28.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:28.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:28.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:28.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:28.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:28.904 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:28.715+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:29.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:28.903+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:29.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:28.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:29.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:28.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:29.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:28.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:29.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:28.916+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:29.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:29.136+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.137522+0000) 2024-02-19T20:24:29.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:29.323+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.324412+0000) 2024-02-19T20:24:29.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:29.587+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:29.588486+0000) 2024-02-19T20:24:29.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:29.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:29.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:29.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:29.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:29.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:29.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:29.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:30.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:29.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:30.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:29.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:30.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:29.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:30.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:29.889+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:30.139 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:29.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:30.139 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:29.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:30.139 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:29.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:30.139 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:29.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:30.539 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:30.325+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.326538+0000) 2024-02-19T20:24:30.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:30.140+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.141140+0000) 2024-02-19T20:24:30.866 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:30.589+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:30.591004+0000) 2024-02-19T20:24:30.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:30.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:30.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:30.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:30.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:30.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:30.867 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:30.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:31.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:30.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:31.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:30.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:31.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:30.868+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:31.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:30.869+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:31.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:30.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:31.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:30.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:31.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:30.949+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:31.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:30.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:31.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:31.328+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.329770+0000) 2024-02-19T20:24:31.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:31.142+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.143245+0000) 2024-02-19T20:24:31.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:31.591+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:31.593007+0000) 2024-02-19T20:24:31.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:31.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:31.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:31.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:31.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:31.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:31.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:31.715+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:32.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:31.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:32.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:31.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:32.143 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:31.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:32.143 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:31.880+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:32.143 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:31.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:32.143 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:31.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:32.143 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:31.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:32.143 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:31.924+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:32.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:32.331+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.332147+0000) 2024-02-19T20:24:32.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:32.144+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.145432+0000) 2024-02-19T20:24:32.854 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:32.737+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62298 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:32.855 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:32.734+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.735720+0000) 2024-02-19T20:24:32.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:32.736+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.737159+0000) 2024-02-19T20:24:32.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:32.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:32.855 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:32.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:32.856 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:32.837+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:32.856 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:32.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:32.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:32.593+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.594522+0000) 2024-02-19T20:24:32.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:32.735+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:32.736407+0000) 2024-02-19T20:24:32.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:32.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:32.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:32.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:32.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:32.746+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:32.857 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:32.748+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:33.142 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:32.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:33.143 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:32.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:33.143 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:32.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:33.143 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:32.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:33.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:33.331+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.332345+0000) 2024-02-19T20:24:33.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:33.144+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.145678+0000) 2024-02-19T20:24:33.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:33.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:33.906 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:33.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:33.907 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:33.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:33.907 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:33.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:33.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:33.593+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:33.594720+0000) 2024-02-19T20:24:33.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:33.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:33.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:33.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:33.907 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:33.740+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:33.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:33.741+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:34.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:34.146+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.147911+0000) 2024-02-19T20:24:34.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:33.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:34.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:33.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:34.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:33.908+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:34.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:33.909+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:34.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:34.333+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.334570+0000) 2024-02-19T20:24:34.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:34.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:34.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:34.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:34.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:34.803+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:34.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:34.804+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:34.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:34.596+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:34.597237+0000) 2024-02-19T20:24:34.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:34.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:34.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:34.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:34.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:34.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:34.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:34.705+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:35.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:35.148+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.149946+0000) 2024-02-19T20:24:35.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:34.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:35.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:34.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:35.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:34.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:35.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:34.928+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:35.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:35.335+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.336867+0000) 2024-02-19T20:24:35.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:35.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:35.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:35.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:35.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:35.767+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:35.919 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:35.768+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:35.919 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:35.598+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:35.599505+0000) 2024-02-19T20:24:35.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:35.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:35.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:35.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:35.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:35.662+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:35.920 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:35.663+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:36.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:36.152+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.153613+0000) 2024-02-19T20:24:36.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:35.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:36.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:35.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:36.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:35.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:36.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:35.953+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:36.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:36.337+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.338903+0000) 2024-02-19T20:24:36.916 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:36.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:36.917 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:36.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:36.917 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:36.744+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:36.917 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:36.744+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:36.917 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:36.600+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:36.601961+0000) 2024-02-19T20:24:36.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:36.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:36.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:36.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:36.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:36.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:36.918 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:36.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:37.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:37.154+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.155028+0000) 2024-02-19T20:24:37.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:36.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:37.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:36.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:37.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:36.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:37.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:36.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:37.601 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:37.340+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.341186+0000) 2024-02-19T20:24:37.867 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:37.746+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62338 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:37.867 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:37.742+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.743693+0000) 2024-02-19T20:24:37.867 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:37.743+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.744925+0000) 2024-02-19T20:24:37.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:37.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:37.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:37.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:37.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:37.749+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:37.868 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:37.750+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:37.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:37.603+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.604313+0000) 2024-02-19T20:24:37.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:37.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:37.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:37.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:37.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:37.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:37.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:37.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:37.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:37.741+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:37.742755+0000) 2024-02-19T20:24:38.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:37.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:38.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:37.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:38.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:37.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:38.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:37.874+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:38.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:38.340+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.341391+0000) 2024-02-19T20:24:38.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:38.154+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.155206+0000) 2024-02-19T20:24:38.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:38.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:38.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:38.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:38.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:38.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:38.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:38.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:38.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:38.603+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:38.604538+0000) 2024-02-19T20:24:38.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:38.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:38.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:38.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:38.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:38.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:38.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:38.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:38.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:38.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:38.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:38.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:38.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:38.912+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:39.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:39.157+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.158062+0000) 2024-02-19T20:24:39.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:38.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:39.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:39.342+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.343775+0000) 2024-02-19T20:24:39.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:39.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:39.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:39.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:39.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:39.809+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:39.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:39.809+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:39.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:39.605+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:39.606837+0000) 2024-02-19T20:24:39.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:39.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:39.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:39.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:39.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:39.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:39.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:39.645+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:40.234 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:40.159+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.160233+0000) 2024-02-19T20:24:40.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:39.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:40.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:39.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:40.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:39.953+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:40.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:39.954+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:40.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:40.346+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.347300+0000) 2024-02-19T20:24:40.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:40.608+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:40.609036+0000) 2024-02-19T20:24:40.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:40.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:40.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:40.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:40.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:40.672+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:40.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:40.673+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:40.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:40.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:40.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:40.798+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:40.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:40.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:40.924 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:40.799+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:41.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:41.161+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.162367+0000) 2024-02-19T20:24:41.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:41.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:41.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:41.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:41.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.349+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.350940+0000) 2024-02-19T20:24:41.922 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:41.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:41.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:41.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:41.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:41.772+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:41.923 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:41.773+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:41.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:41.610+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:41.611476+0000) 2024-02-19T20:24:41.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:41.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:41.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:41.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:41.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:41.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:41.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:41.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:42.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:42.162+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.163953+0000) 2024-02-19T20:24:42.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:42.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:42.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.993+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:42.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:41.994+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:42.609 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:42.352+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.353311+0000) 2024-02-19T20:24:42.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:42.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:42.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:42.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:42.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:42.597+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:42.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:42.598+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:42.869 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:42.753+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62377 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:42.870 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:42.750+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.751316+0000) 2024-02-19T20:24:42.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:42.612+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.613093+0000) 2024-02-19T20:24:42.870 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:42.749+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.750919+0000) 2024-02-19T20:24:42.870 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:42.750+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:42.751732+0000) 2024-02-19T20:24:42.870 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:42.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:42.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:42.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:42.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:42.791+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:42.871 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:42.791+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:43.160 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:42.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:43.161 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:42.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:43.161 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:42.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:43.161 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:42.969+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:43.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:43.352+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.353485+0000) 2024-02-19T20:24:43.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:43.163+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.164150+0000) 2024-02-19T20:24:43.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:43.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:43.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:43.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:43.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:43.840+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:43.920 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:43.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:43.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:43.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:43.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:43.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:43.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:43.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:43.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:43.606+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:43.921 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:43.612+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:43.613259+0000) 2024-02-19T20:24:44.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:44.166+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.167490+0000) 2024-02-19T20:24:44.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:43.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:44.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:43.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:44.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:43.922+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:44.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:43.923+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:44.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:44.355+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.356048+0000) 2024-02-19T20:24:44.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:44.614+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:44.615331+0000) 2024-02-19T20:24:44.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:44.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:44.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:44.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:44.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:44.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:44.926 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:44.634+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:44.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:44.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:44.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:44.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:44.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:44.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:44.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:44.849+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:45.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:45.169+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.170909+0000) 2024-02-19T20:24:45.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:44.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:45.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:44.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:45.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:44.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:45.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:44.957+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:45.614 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:45.358+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.359445+0000) 2024-02-19T20:24:45.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:45.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:45.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:45.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:45.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:45.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:45.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:45.833+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:45.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:45.616+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:45.617439+0000) 2024-02-19T20:24:45.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:45.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:45.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:45.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:45.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:45.677+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:45.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:45.678+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:46.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:46.173+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.174363+0000) 2024-02-19T20:24:46.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:45.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:46.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:45.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:46.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:45.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:46.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:45.942+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:46.616 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:46.361+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.362130+0000) 2024-02-19T20:24:46.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:46.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:46.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:46.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:46.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:46.792+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:46.926 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:46.793+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:46.618+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:46.619545+0000) 2024-02-19T20:24:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:46.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:46.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:46.653+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:46.927 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:46.654+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:47.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:47.175+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.176724+0000) 2024-02-19T20:24:47.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:46.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:47.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:46.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:47.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:46.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:47.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:46.955+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:47.617 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:47.362+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.363691+0000) 2024-02-19T20:24:47.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:47.757+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.758918+0000) 2024-02-19T20:24:47.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:47.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:47.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:47.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:47.878 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:47.832+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:47.879 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:47.833+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:47.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:47.620+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.621173+0000) 2024-02-19T20:24:47.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:47.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:47.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:47.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:47.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:47.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:47.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:47.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:47.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:47.757+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.758341+0000) 2024-02-19T20:24:47.880 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:47.760+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62417 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:47.880 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:47.758+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:47.759614+0000) 2024-02-19T20:24:48.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:47.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:48.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:47.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:48.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:47.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:48.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:47.982+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:48.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:48.362+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.363863+0000) 2024-02-19T20:24:48.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:48.175+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.176952+0000) 2024-02-19T20:24:48.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:48.620+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:48.621350+0000) 2024-02-19T20:24:48.868 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:48.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:48.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:48.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:48.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:48.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:48.869 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:48.672+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:49.175 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:48.870+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:49.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:48.870+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:49.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:48.870+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:49.176 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:48.871+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:49.176 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:48.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:49.176 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:48.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:49.177 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:48.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:49.177 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:48.968+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:49.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:49.178+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.179225+0000) 2024-02-19T20:24:49.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:49.365+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.366280+0000) 2024-02-19T20:24:49.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:49.623+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:49.624798+0000) 2024-02-19T20:24:49.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:49.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:49.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:49.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:49.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:49.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:49.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:49.667+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:50.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:49.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:50.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:49.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:50.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:49.849+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:50.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:49.850+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:50.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:49.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:50.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:49.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:50.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:49.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:50.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:49.983+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:50.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:50.180+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.181605+0000) 2024-02-19T20:24:50.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:50.367+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.368674+0000) 2024-02-19T20:24:50.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:50.627+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:50.628251+0000) 2024-02-19T20:24:50.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:50.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:50.879 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:50.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:50.880 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:50.688+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:50.880 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:50.689+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:51.180 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:51.181 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:51.181 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:51.181 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.010+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:51.181 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:50.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:51.181 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:50.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:51.181 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:50.881+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:51.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:50.882+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:51.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:51.183+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.184222+0000) 2024-02-19T20:24:51.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.370+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.371120+0000) 2024-02-19T20:24:51.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:51.630+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:51.631699+0000) 2024-02-19T20:24:51.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:51.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:51.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:51.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:51.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:51.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:51.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:51.697+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:52.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:52.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:52.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.980+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:52.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:51.981+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:52.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:51.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:52.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:51.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:52.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:51.848+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:52.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:51.848+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:52.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:52.184+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.185972+0000) 2024-02-19T20:24:52.630 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:52.371+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.372902+0000) 2024-02-19T20:24:52.884 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:52.768+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62456 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:52.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:52.766+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.767324+0000) 2024-02-19T20:24:52.885 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:52.765+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.766025+0000) 2024-02-19T20:24:52.885 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:52.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:52.886 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:52.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:52.886 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:52.852+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:52.886 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:52.853+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:52.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:52.632+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.633984+0000) 2024-02-19T20:24:52.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:52.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:52.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:52.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:52.886 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:52.665+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:52.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:52.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:52.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:52.765+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:52.766224+0000) 2024-02-19T20:24:53.182 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:53.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:53.183 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:53.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:53.183 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:53.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:53.183 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:53.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:53.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:53.372+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.373140+0000) 2024-02-19T20:24:53.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:53.185+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.186169+0000) 2024-02-19T20:24:53.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:53.634+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:53.634225+0000) 2024-02-19T20:24:53.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:53.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:53.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:53.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:53.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:53.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:53.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:53.635+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:54.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:53.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:54.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:53.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:54.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:53.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:54.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:53.856+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:54.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:54.188+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.188788+0000) 2024-02-19T20:24:54.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:54.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:54.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:54.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:54.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:54.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:54.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:54.042+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:54.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:54.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:54.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:54.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:54.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:54.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:54.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:54.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:54.634 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:54.376+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.376723+0000) 2024-02-19T20:24:54.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:54.635+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:54.636568+0000) 2024-02-19T20:24:54.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:54.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:54.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:54.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:54.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:54.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:54.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:54.821+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:55.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:55.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:55.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:55.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:55.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:55.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:55.188 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:55.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:55.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:55.190+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.191331+0000) 2024-02-19T20:24:55.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:55.378+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.379303+0000) 2024-02-19T20:24:55.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:55.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:55.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:55.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:55.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:55.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:55.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:55.594+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:55.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:55.637+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:55.638743+0000) 2024-02-19T20:24:56.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:55.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:56.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:55.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:56.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:55.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:56.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:55.854+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:56.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:56.192+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.193671+0000) 2024-02-19T20:24:56.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:56.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:56.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:56.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:56.378 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:56.092+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:56.379 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:56.093+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:56.640 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:56.380+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.381771+0000) 2024-02-19T20:24:56.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:56.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:56.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:56.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:56.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:56.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:56.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:56.574+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:56.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:56.639+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:56.640756+0000) 2024-02-19T20:24:56.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:56.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:56.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:56.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:56.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:56.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:56.937 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:56.860+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:57.191 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:57.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:57.191 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:57.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:57.192 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:57.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:57.192 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:57.050+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:57.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:57.383+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.384127+0000) 2024-02-19T20:24:57.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:57.194+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.195127+0000) 2024-02-19T20:24:57.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:57.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:57.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:57.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:57.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:57.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:57.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:57.607+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:57.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:57.641+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.642479+0000) 2024-02-19T20:24:58.034 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:24:57.776+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62496 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:24:58.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:57.772+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.773987+0000) 2024-02-19T20:24:58.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:57.773+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.774790+0000) 2024-02-19T20:24:58.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:57.772+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:57.773986+0000) 2024-02-19T20:24:58.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:57.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:58.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:57.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:58.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:57.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:58.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:57.866+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:58.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:58.194+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.195306+0000) 2024-02-19T20:24:58.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:58.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:58.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:58.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:58.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:58.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:58.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:58.037+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:58.639 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:58.383+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.384440+0000) 2024-02-19T20:24:58.939 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:58.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:58.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:58.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:58.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:58.883+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:24:58.940 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:58.883+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:58.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:58.641+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:58.642674+0000) 2024-02-19T20:24:58.940 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:58.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:58.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:58.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:58.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:58.645+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:58.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:58.645+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:59.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:59.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:59.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:59.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:59.194 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:59.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:24:59.195 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:59.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:24:59.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:24:59.387+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.388066+0000) 2024-02-19T20:24:59.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:59.196+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.197742+0000) 2024-02-19T20:24:59.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:59.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:59.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:59.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:59.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:59.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:24:59.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:59.627+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:24:59.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:24:59.643+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:24:59.644848+0000) 2024-02-19T20:25:00.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:59.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:00.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:59.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:00.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:59.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:00.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:24:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:24:59.844+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:00.386 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:00.198+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.199924+0000) 2024-02-19T20:25:00.387 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:00.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:00.387 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:00.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:00.387 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:00.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:00.387 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:00.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:00.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:00.389+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.390082+0000) 2024-02-19T20:25:00.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:00.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:00.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:00.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:00.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:00.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:00.941 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:00.889+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:00.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:00.645+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:00.646878+0000) 2024-02-19T20:25:00.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:00.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:00.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:00.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:00.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:00.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:00.942 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:00.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:01.198 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:01.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:01.199 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:01.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:01.199 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:01.099+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:01.199 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:01.100+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:01.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:01.200+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.201968+0000) 2024-02-19T20:25:01.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:01.391+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.392287+0000) 2024-02-19T20:25:01.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:01.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:01.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:01.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:01.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:01.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:01.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:01.644+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:01.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:01.649+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:01.650509+0000) 2024-02-19T20:25:02.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:01.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:02.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:01.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:02.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:01.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:02.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:01.864+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:02.349 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:02.202+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.203808+0000) 2024-02-19T20:25:02.349 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:02.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:02.350 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:02.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:02.350 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:02.141+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:02.350 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:02.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:02.648 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:02.392+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.393753+0000) 2024-02-19T20:25:02.900 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:02.780+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.781558+0000) 2024-02-19T20:25:02.900 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:02.784+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62535 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:02.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:02.779+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.780992+0000) 2024-02-19T20:25:02.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:02.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:02.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:02.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:02.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:02.836+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:02.901 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:02.837+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:02.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:02.650+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.651920+0000) 2024-02-19T20:25:02.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:02.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:02.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:02.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:02.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:02.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:02.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:02.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:02.902 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:02.781+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:02.782091+0000) 2024-02-19T20:25:03.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:03.202+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.204004+0000) 2024-02-19T20:25:03.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:03.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:03.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:03.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:03.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:03.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:03.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:03.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:03.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:03.392+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.393959+0000) 2024-02-19T20:25:03.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:03.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:03.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:03.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:03.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:03.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:03.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:03.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:03.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:03.651+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:03.652052+0000) 2024-02-19T20:25:04.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:03.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:04.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:03.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:04.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:03.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:04.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:03.822+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:04.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:04.205+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.206063+0000) 2024-02-19T20:25:04.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:04.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:04.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:04.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:04.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:04.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:04.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:04.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:04.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:04.395+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.396049+0000) 2024-02-19T20:25:04.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:04.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:04.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:04.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:04.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:04.843+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:04.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:04.844+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:04.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:04.653+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:04.654123+0000) 2024-02-19T20:25:04.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:04.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:04.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:04.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:04.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:04.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:04.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:04.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:05.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:05.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:05.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:05.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:05.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:05.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:05.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:05.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:05.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:05.208+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.209690+0000) 2024-02-19T20:25:05.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:05.397+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.398179+0000) 2024-02-19T20:25:05.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:05.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:05.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:05.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:05.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:05.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:05.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:05.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:05.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:05.655+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:05.656647+0000) 2024-02-19T20:25:06.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:05.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:06.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:05.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:06.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:05.799+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:06.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:05.800+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:06.398 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:06.210+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.211929+0000) 2024-02-19T20:25:06.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:06.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:06.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:06.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:06.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:06.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:06.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:06.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:06.655 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:06.400+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.401782+0000) 2024-02-19T20:25:06.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:06.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:06.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:06.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:06.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:06.616+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:06.656 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:06.617+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:06.946 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:06.658+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:06.659438+0000) 2024-02-19T20:25:06.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:06.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:06.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:06.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:06.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:06.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:06.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:06.794+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:07.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:07.212+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.213361+0000) 2024-02-19T20:25:07.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:07.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:07.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:07.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:07.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:07.206+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:07.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:07.207+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:07.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:07.402+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.403555+0000) 2024-02-19T20:25:07.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:07.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:07.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:07.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:07.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:07.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:07.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:07.631+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:07.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:07.660+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.661131+0000) 2024-02-19T20:25:08.038 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:07.791+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62575 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:08.038 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:07.789+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.790237+0000) 2024-02-19T20:25:08.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:07.788+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.789678+0000) 2024-02-19T20:25:08.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:07.789+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:07.790310+0000) 2024-02-19T20:25:08.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:07.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:08.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:07.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:08.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:07.826+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:08.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:07.827+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:08.400 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:08.212+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.213525+0000) 2024-02-19T20:25:08.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:08.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:08.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:08.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:08.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:08.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:08.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:08.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:08.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:08.402+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.403779+0000) 2024-02-19T20:25:08.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:08.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:08.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:08.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:08.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:08.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:08.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:08.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:08.949 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:08.660+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:08.661364+0000) 2024-02-19T20:25:08.949 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:08.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:08.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:08.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:08.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:08.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:08.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:08.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:09.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:09.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:09.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:09.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:09.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:09.200+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:09.212 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:09.200+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:09.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:09.406+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.407167+0000) 2024-02-19T20:25:09.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:09.214+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.216000+0000) 2024-02-19T20:25:09.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:09.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:09.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:09.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:09.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:09.598+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:09.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:09.599+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:09.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:09.662+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:09.663361+0000) 2024-02-19T20:25:10.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:09.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:10.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:09.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:10.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:09.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:10.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:09.834+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:10.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:10.217+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.218393+0000) 2024-02-19T20:25:10.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:10.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:10.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:10.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:10.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:10.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:10.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:10.177+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:10.407 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:10.408+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.409398+0000) 2024-02-19T20:25:10.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:10.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:10.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:10.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:10.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:10.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:10.662 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:10.632+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:10.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:10.664+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:10.665478+0000) 2024-02-19T20:25:10.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:10.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:10.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:10.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:10.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:10.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:10.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:10.797+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:11.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:11.219+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.220494+0000) 2024-02-19T20:25:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:11.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:11.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:11.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:11.216+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:11.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:11.410+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.411659+0000) 2024-02-19T20:25:11.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:11.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:11.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:11.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:11.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:11.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:11.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:11.602+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:11.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:11.666+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:11.667658+0000) 2024-02-19T20:25:12.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:11.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:12.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:11.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:12.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:11.823+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:12.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:11.823+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:12.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:12.220+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.221976+0000) 2024-02-19T20:25:12.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:12.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:12.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:12.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:12.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:12.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:12.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:12.184+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:12.665 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:12.412+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.413534+0000) 2024-02-19T20:25:12.665 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:12.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:12.665 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:12.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:12.665 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:12.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:12.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:12.614+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:12.951 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:12.799+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62614 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:12.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:12.795+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.796787+0000) 2024-02-19T20:25:12.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:12.668+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.669086+0000) 2024-02-19T20:25:12.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:12.796+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.797446+0000) 2024-02-19T20:25:12.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:12.796+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:12.797219+0000) 2024-02-19T20:25:12.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:12.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:12.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:12.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:12.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:12.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:12.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:12.867+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:13.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:13.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:13.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:13.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:13.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:13.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:13.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:13.169+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:13.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:13.412+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.413734+0000) 2024-02-19T20:25:13.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:13.221+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.222200+0000) 2024-02-19T20:25:13.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:13.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:13.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:13.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:13.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:13.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:13.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:13.614+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:13.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:13.668+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:13.669319+0000) 2024-02-19T20:25:14.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:13.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:14.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:13.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:14.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:13.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:14.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:13.830+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:14.412 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:14.223+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.224617+0000) 2024-02-19T20:25:14.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:14.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:14.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:14.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:14.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:14.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:14.413 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:14.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:14.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:14.415+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.416296+0000) 2024-02-19T20:25:14.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:14.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:14.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:14.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:14.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:14.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:14.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:14.578+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:14.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:14.670+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:14.671681+0000) 2024-02-19T20:25:14.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:14.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:14.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:14.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:14.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:14.796+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:14.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:14.797+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:15.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:15.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:15.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:15.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:15.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:15.129+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:15.223 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:15.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:15.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:15.417+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.418392+0000) 2024-02-19T20:25:15.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:15.225+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.226682+0000) 2024-02-19T20:25:15.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:15.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:15.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:15.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:15.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:15.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:15.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:15.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:15.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:15.673+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:15.674131+0000) 2024-02-19T20:25:16.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:15.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:16.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:15.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:16.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:15.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:16.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:15.816+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:16.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:16.228+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.229089+0000) 2024-02-19T20:25:16.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:16.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:16.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:16.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:16.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:16.150+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:16.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:16.151+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:16.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:16.419+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.420648+0000) 2024-02-19T20:25:16.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:16.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:16.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:16.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:16.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:16.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:16.673 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:16.613+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:16.955 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:16.675+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:16.676476+0000) 2024-02-19T20:25:16.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:16.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:16.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:16.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:16.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:16.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:16.956 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:16.852+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:17.226 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:17.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:17.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:17.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:17.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:17.114+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:17.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:17.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:17.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:17.422+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.423084+0000) 2024-02-19T20:25:17.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:17.229+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.230813+0000) 2024-02-19T20:25:17.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:17.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:17.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:17.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:17.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:17.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:17.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:17.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:17.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:17.677+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.678123+0000) 2024-02-19T20:25:18.037 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:17.806+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62654 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:18.038 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:17.803+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.804156+0000) 2024-02-19T20:25:18.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:17.805+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.806126+0000) 2024-02-19T20:25:18.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:17.804+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:17.805053+0000) 2024-02-19T20:25:18.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:17.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:18.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:17.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:18.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:17.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:18.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:17.812+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:18.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:18.230+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.231032+0000) 2024-02-19T20:25:18.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:18.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:18.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:18.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:18.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:18.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:18.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:18.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:18.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:18.422+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.423245+0000) 2024-02-19T20:25:18.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:18.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:18.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:18.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:18.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:18.588+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:18.675 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:18.589+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:18.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:18.677+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:18.678357+0000) 2024-02-19T20:25:18.958 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:18.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:18.958 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:18.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:18.958 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:18.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:18.958 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:18.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:19.232 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:19.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:19.232 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:19.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:19.232 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:19.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:19.233 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:19.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:19.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:19.232+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.233822+0000) 2024-02-19T20:25:19.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:19.425+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.426489+0000) 2024-02-19T20:25:19.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:19.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:19.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:19.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:19.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:19.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:19.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:19.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:19.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:19.679+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:19.680999+0000) 2024-02-19T20:25:20.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:19.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:20.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:19.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:20.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:19.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:20.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:19.842+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:20.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:20.235+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.236053+0000) 2024-02-19T20:25:20.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:20.066+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:20.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:20.066+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:20.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:20.066+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:20.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:20.067+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:20.679 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:20.428+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.429864+0000) 2024-02-19T20:25:20.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:20.648+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:20.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:20.648+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:20.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:20.648+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:20.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:20.649+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:20.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:20.682+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:20.683058+0000) 2024-02-19T20:25:20.959 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:20.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:20.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:20.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:20.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:20.795+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:20.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:20.795+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:21.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:21.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:21.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:21.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:21.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:21.054+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:21.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:21.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:21.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:21.432+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.433260+0000) 2024-02-19T20:25:21.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:21.237+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.238063+0000) 2024-02-19T20:25:21.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:21.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:21.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:21.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:21.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:21.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:21.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:21.677+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:21.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:21.684+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:21.685197+0000) 2024-02-19T20:25:22.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:21.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:22.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:21.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:22.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:21.794+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:22.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:21.795+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:22.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:22.238+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.239909+0000) 2024-02-19T20:25:22.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:22.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:22.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:22.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:22.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:22.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:22.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:22.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:22.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:22.434+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.435749+0000) 2024-02-19T20:25:22.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:22.813+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62693 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:22.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:22.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:22.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:22.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:22.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:22.805+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:22.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:22.805+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:22.961 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:22.810+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.811249+0000) 2024-02-19T20:25:22.961 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:22.811+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.812990+0000) 2024-02-19T20:25:22.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:22.685+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.686973+0000) 2024-02-19T20:25:22.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:22.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:22.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:22.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:22.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:22.718+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:22.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:22.719+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:22.962 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:22.811+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:22.812139+0000) 2024-02-19T20:25:23.235 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:23.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:23.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:23.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:23.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:23.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:23.236 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:23.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:23.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:23.434+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.435961+0000) 2024-02-19T20:25:23.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:23.239+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.240118+0000) 2024-02-19T20:25:23.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:23.686+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:23.687086+0000) 2024-02-19T20:25:23.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:23.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:23.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:23.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:23.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:23.705+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:23.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:23.706+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:24.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:23.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:24.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:23.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:24.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:23.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:24.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:23.842+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:24.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:24.241+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.242714+0000) 2024-02-19T20:25:24.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:24.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:24.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:24.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:24.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:24.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:24.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:24.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:24.685 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:24.437+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.438153+0000) 2024-02-19T20:25:24.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:24.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:24.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:24.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:24.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:24.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:24.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:24.684+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:24.686 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:24.688+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:24.689214+0000) 2024-02-19T20:25:24.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:24.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:24.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:24.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:24.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:24.833+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:24.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:24.833+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:25.241 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:25.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:25.241 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:25.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:25.241 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:25.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:25.241 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:25.028+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:25.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:25.439+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.440741+0000) 2024-02-19T20:25:25.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:25.243+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.244765+0000) 2024-02-19T20:25:25.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:25.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:25.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:25.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:25.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:25.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:25.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:25.676+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:25.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:25.690+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:25.691530+0000) 2024-02-19T20:25:26.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:25.863+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:26.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:25.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:26.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:25.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:26.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:25.864+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:26.438 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:26.245+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.246947+0000) 2024-02-19T20:25:26.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:26.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:26.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:26.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:26.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:26.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:26.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:26.054+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:26.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:26.441+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.442779+0000) 2024-02-19T20:25:26.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:26.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:26.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:26.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:26.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:26.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:26.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:26.684+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:26.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:26.692+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:26.693972+0000) 2024-02-19T20:25:26.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:26.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:26.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:26.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:26.964 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:26.862+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:26.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:26.862+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:27.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:27.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:27.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:27.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:27.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:27.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:27.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:27.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:27.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:27.443+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.444356+0000) 2024-02-19T20:25:27.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:27.248+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.249452+0000) 2024-02-19T20:25:27.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:27.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:27.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:27.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:27.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:27.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:27.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:27.670+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:27.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:27.694+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.695571+0000) 2024-02-19T20:25:28.040 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:27.821+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62733 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:28.041 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:27.817+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.818942+0000) 2024-02-19T20:25:28.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:27.817+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.818500+0000) 2024-02-19T20:25:28.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:27.817+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:27.818805+0000) 2024-02-19T20:25:28.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:27.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:27.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:27.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:28.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:27.871+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:28.440 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:28.248+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.249653+0000) 2024-02-19T20:25:28.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:28.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:28.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:28.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:28.441 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:28.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:28.441 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:28.104+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:28.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:28.443+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.444539+0000) 2024-02-19T20:25:28.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:28.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:28.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:28.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:28.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:28.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:28.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:28.631+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:28.964 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:28.694+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:28.695786+0000) 2024-02-19T20:25:28.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:28.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:28.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:28.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:28.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:28.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:28.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:28.868+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:29.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:29.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:29.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:29.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:29.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:29.106+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:29.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:29.107+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:29.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:29.446+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.447134+0000) 2024-02-19T20:25:29.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:29.251+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.252232+0000) 2024-02-19T20:25:29.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:29.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:29.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:29.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:29.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:29.644+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:29.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:29.645+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:29.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:29.697+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:29.698331+0000) 2024-02-19T20:25:30.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:29.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:30.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:29.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:30.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:29.824+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:30.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:29.825+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:30.446 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:30.254+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.255801+0000) 2024-02-19T20:25:30.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:30.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:30.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:30.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:30.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:30.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:30.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:30.125+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:30.697 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:30.449+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.450726+0000) 2024-02-19T20:25:30.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:30.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:30.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:30.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:30.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:30.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:30.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:30.669+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:30.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:30.701+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:30.702065+0000) 2024-02-19T20:25:30.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:30.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:30.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:30.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:30.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:30.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:30.966 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:30.865+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:31.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:31.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:31.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:31.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:31.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:31.145+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:31.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:31.146+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:31.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:31.257+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.258018+0000) 2024-02-19T20:25:31.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:31.452+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.453190+0000) 2024-02-19T20:25:31.887 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:31.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:31.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:31.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:31.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:31.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:31.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:31.683+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:31.888 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:31.703+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:31.704276+0000) 2024-02-19T20:25:32.255 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:31.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:32.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:31.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:32.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:31.891+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:32.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:31.891+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:32.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:32.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:32.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:32.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:32.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:32.110+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:32.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:32.111+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:32.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:32.258+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.259915+0000) 2024-02-19T20:25:32.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:32.453+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.454690+0000) 2024-02-19T20:25:32.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:32.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:32.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:32.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:32.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:32.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:32.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:32.701+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:32.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:32.705+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.706490+0000) 2024-02-19T20:25:33.070 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:32.829+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62772 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:33.071 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:32.825+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.826858+0000) 2024-02-19T20:25:33.071 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:32.824+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.825981+0000) 2024-02-19T20:25:33.071 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:32.825+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:32.826484+0000) 2024-02-19T20:25:33.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:32.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:33.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:32.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:33.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:32.871+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:33.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:32.872+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:33.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:33.259+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.260113+0000) 2024-02-19T20:25:33.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:33.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:33.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:33.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:33.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:33.084+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:33.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:33.084+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:33.702 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:33.453+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.454885+0000) 2024-02-19T20:25:33.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:33.895+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:33.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:33.895+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:33.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:33.895+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:33.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:33.895+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:33.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:33.705+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:33.706722+0000) 2024-02-19T20:25:33.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:33.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:33.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:33.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:33.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:33.720+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:33.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:33.721+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:34.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:34.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:34.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:34.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:34.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:34.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:34.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:34.050+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:34.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:34.456+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.457713+0000) 2024-02-19T20:25:34.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:34.261+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.262120+0000) 2024-02-19T20:25:34.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:34.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:34.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:34.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:34.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:34.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:34.877 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:34.676+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:34.878 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:34.707+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:34.708910+0000) 2024-02-19T20:25:35.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:34.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:35.262 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:34.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:35.262 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:34.880+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:35.262 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:34.881+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:35.263 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:35.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:35.263 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:35.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:35.263 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:35.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:35.263 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:35.068+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:35.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:35.458+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.459777+0000) 2024-02-19T20:25:35.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:35.264+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.265768+0000) 2024-02-19T20:25:35.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:35.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:35.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:35.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:35.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:35.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:35.912 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:35.669+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:35.912 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:35.709+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:35.710854+0000) 2024-02-19T20:25:36.263 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:35.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:36.264 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:35.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:36.264 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:35.914+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:36.264 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:35.915+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:36.264 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:36.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:36.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:36.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:36.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:36.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:36.265 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:36.028+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:36.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:36.461+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.462017+0000) 2024-02-19T20:25:36.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:36.267+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.268122+0000) 2024-02-19T20:25:36.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:36.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:36.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:36.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:36.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:36.627+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:36.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:36.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:36.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:36.711+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:36.712980+0000) 2024-02-19T20:25:37.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:37.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:37.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:37.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:37.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:37.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:37.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:37.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:37.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:36.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:37.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:36.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:37.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:36.937+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:37.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:36.938+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:37.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:37.269+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.270370+0000) 2024-02-19T20:25:37.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:37.462+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.463711+0000) 2024-02-19T20:25:37.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:37.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:37.827 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:37.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:37.827 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:37.613+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:37.827 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:37.614+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:37.827 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:37.713+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.714474+0000) 2024-02-19T20:25:38.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:37.837+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62812 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:38.266 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:37.832+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.833880+0000) 2024-02-19T20:25:38.266 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:37.833+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.834644+0000) 2024-02-19T20:25:38.267 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:38.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:38.267 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:38.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:38.267 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:38.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:38.267 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:38.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:38.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:37.833+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:37.834987+0000) 2024-02-19T20:25:38.267 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:37.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:38.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:37.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:38.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:37.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:38.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:37.929+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:38.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:38.462+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.463939+0000) 2024-02-19T20:25:38.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:38.269+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.270561+0000) 2024-02-19T20:25:38.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:38.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:38.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:38.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:38.938 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:38.606+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:38.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:38.607+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:38.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:38.713+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:38.714664+0000) 2024-02-19T20:25:39.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:38.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:39.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:38.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:39.270 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:38.941+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:39.270 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:38.942+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:39.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:38.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:39.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:38.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:39.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:38.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:39.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:39.000+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:39.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:39.464+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.465981+0000) 2024-02-19T20:25:39.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:39.272+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.273818+0000) 2024-02-19T20:25:39.896 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:39.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:39.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:39.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:39.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:39.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:39.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:39.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:39.897 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:39.715+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:39.716693+0000) 2024-02-19T20:25:40.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:39.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:40.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:39.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:40.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:39.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:40.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:39.966+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:40.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:39.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:40.239 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:39.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:40.239 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:39.901+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:40.239 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:39.901+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:40.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:40.468+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.468150+0000) 2024-02-19T20:25:40.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:40.277+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.277200+0000) 2024-02-19T20:25:40.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:40.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:40.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:40.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:40.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:40.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:40.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:40.554+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:40.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:40.718+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:40.718866+0000) 2024-02-19T20:25:41.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:40.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:41.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:40.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:41.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:40.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:41.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:40.939+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:41.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:40.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:41.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:40.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:41.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:40.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:41.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:40.928+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:41.467 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:41.278+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.279223+0000) 2024-02-19T20:25:41.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:41.470+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.471983+0000) 2024-02-19T20:25:41.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:41.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:41.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:41.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:41.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:41.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:41.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:41.557+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:41.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:41.719+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:41.720920+0000) 2024-02-19T20:25:42.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:41.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:42.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:41.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:42.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:41.968+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:42.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:41.968+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:42.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:41.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:42.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:41.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:42.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:41.965+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:42.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:41.965+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:42.469 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:42.280+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.281060+0000) 2024-02-19T20:25:42.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:42.472+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.473726+0000) 2024-02-19T20:25:42.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:42.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:42.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:42.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:42.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:42.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:42.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:42.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:42.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:42.721+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.722394+0000) 2024-02-19T20:25:43.037 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:42.844+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62851 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:43.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:42.841+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.842848+0000) 2024-02-19T20:25:43.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:42.842+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.843301+0000) 2024-02-19T20:25:43.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:42.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:43.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:42.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:43.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:42.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:43.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:42.954+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:43.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:42.842+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:42.843848+0000) 2024-02-19T20:25:43.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:43.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:43.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:43.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.014+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:43.460 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:43.280+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.281272+0000) 2024-02-19T20:25:43.718 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.472+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.473950+0000) 2024-02-19T20:25:43.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:43.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:43.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:43.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:43.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:43.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:43.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:43.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:43.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:43.721+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:43.722642+0000) 2024-02-19T20:25:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:44.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:43.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:44.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:43.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:44.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:43.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:44.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:43.978+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:44.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:43.978+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:44.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:44.474+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.475987+0000) 2024-02-19T20:25:44.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:44.282+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.283400+0000) 2024-02-19T20:25:44.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:44.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:44.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:44.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:44.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:44.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:44.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:44.486+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:44.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:44.724+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:44.725333+0000) 2024-02-19T20:25:45.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:44.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:45.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:44.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:45.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:44.972+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:45.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:44.972+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:45.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:44.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:45.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:44.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:45.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:44.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:45.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:44.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:45.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:45.477+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.478049+0000) 2024-02-19T20:25:45.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:45.284+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.285668+0000) 2024-02-19T20:25:45.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:45.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:45.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:45.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:45.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:45.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:45.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:45.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:45.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:45.726+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:45.727920+0000) 2024-02-19T20:25:46.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:46.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:46.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:46.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:46.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:46.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:46.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:46.021+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:46.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:45.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:46.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:45.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:46.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:45.974+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:46.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:45.974+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:46.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:46.288+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.289295+0000) 2024-02-19T20:25:46.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:46.479+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.480115+0000) 2024-02-19T20:25:46.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:46.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:46.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:46.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:46.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:46.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:46.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:46.441+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:46.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:46.729+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:46.730629+0000) 2024-02-19T20:25:47.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:47.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:47.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:47.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.024+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:47.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:46.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:47.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:46.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:47.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:46.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:47.289 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:46.999+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:47.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.290+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.291318+0000) 2024-02-19T20:25:47.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:47.481+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.482018+0000) 2024-02-19T20:25:47.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:47.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:47.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:47.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:47.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:47.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:47.538 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:47.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:47.842 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:47.731+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.732261+0000) 2024-02-19T20:25:48.286 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:47.852+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62891 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:48.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:47.849+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.850385+0000) 2024-02-19T20:25:48.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:47.848+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.849766+0000) 2024-02-19T20:25:48.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:47.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:48.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:47.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:48.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:47.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:48.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:47.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:48.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.848+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:47.849917+0000) 2024-02-19T20:25:48.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:48.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:48.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:48.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:47.994+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:48.549 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:48.290+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.291475+0000) 2024-02-19T20:25:48.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:48.481+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.482193+0000) 2024-02-19T20:25:48.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:48.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:48.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:48.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:48.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:48.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:48.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:48.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:48.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:48.731+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:48.732426+0000) 2024-02-19T20:25:49.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:49.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:49.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.002+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:49.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.003+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:49.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:48.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:49.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:48.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:49.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:48.941+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:49.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:48.941+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:49.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:49.483+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.484369+0000) 2024-02-19T20:25:49.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.292+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.293965+0000) 2024-02-19T20:25:49.550 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:49.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:49.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:49.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:49.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:49.452+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:49.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:49.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:49.909 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:49.733+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:49.734389+0000) 2024-02-19T20:25:50.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:50.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:50.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:50.238 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:49.971+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:50.238 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:49.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:50.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:49.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:50.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:49.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:50.239 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:49.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:50.500 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:50.485+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.486540+0000) 2024-02-19T20:25:50.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:50.295+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.296183+0000) 2024-02-19T20:25:50.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:50.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:50.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:50.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:50.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:50.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:50.501 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:50.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:50.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:50.736+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:50.737940+0000) 2024-02-19T20:25:51.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:50.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:51.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:50.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:51.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:50.928+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:51.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:50.928+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:51.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:51.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:51.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.020+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:51.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.021+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:51.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.297+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.298350+0000) 2024-02-19T20:25:51.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:51.487+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.488611+0000) 2024-02-19T20:25:51.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:51.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:51.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:51.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:51.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:51.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:51.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:51.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:51.987 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:51.740+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:51.741552+0000) 2024-02-19T20:25:51.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:51.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:51.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:51.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:51.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:51.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:51.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:51.966+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:52.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:52.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:52.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:52.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:51.991+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:52.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:52.298+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.299904+0000) 2024-02-19T20:25:52.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:52.489+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.490166+0000) 2024-02-19T20:25:52.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:52.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:52.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:52.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:52.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:52.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:52.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:52.378+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:52.850 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:52.742+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.743214+0000) 2024-02-19T20:25:53.287 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:52.860+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62930 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:53.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:52.857+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.858447+0000) 2024-02-19T20:25:53.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:52.856+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.857553+0000) 2024-02-19T20:25:53.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:53.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:53.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.000+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:53.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.001+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:53.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:52.857+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:52.858189+0000) 2024-02-19T20:25:53.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:53.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:53.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:53.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:53.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:53.037+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:53.289 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:53.038+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:53.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:53.299+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.300094+0000) 2024-02-19T20:25:53.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.489+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.490371+0000) 2024-02-19T20:25:53.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:53.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:53.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:53.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:53.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:53.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:53.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:53.405+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:53.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:53.742+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:53.743371+0000) 2024-02-19T20:25:54.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:54.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:54.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:54.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:53.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:54.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:54.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:54.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:54.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:54.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:54.052+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:54.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:54.052+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:54.550 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:54.492+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.493894+0000) 2024-02-19T20:25:54.551 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:54.301+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.302454+0000) 2024-02-19T20:25:54.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:54.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:54.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:54.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:54.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:54.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:54.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:54.425+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:54.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:54.745+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:54.746794+0000) 2024-02-19T20:25:55.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:55.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:55.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:55.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.023+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:55.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:54.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:55.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:54.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:55.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:54.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:55.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:54.980+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:55.550 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.303+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.304844+0000) 2024-02-19T20:25:55.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:55.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:55.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:55.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:55.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:55.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:55.551 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:55.461+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:55.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:55.495+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.496360+0000) 2024-02-19T20:25:55.966 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:55.747+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:55.748997+0000) 2024-02-19T20:25:56.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:56.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:56.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:56.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:55.992+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:56.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:56.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:56.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:56.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:56.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:56.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:56.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:56.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:56.747 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:56.497+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.498419+0000) 2024-02-19T20:25:56.748 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:56.305+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.306857+0000) 2024-02-19T20:25:56.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:56.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:56.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:56.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:56.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:56.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:56.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:56.455+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:57.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:56.751+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:56.752683+0000) 2024-02-19T20:25:57.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:57.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:57.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:57.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:57.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:57.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:57.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:57.037+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:57.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:57.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:57.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.022+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:57.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.022+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:57.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:57.499+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.500479+0000) 2024-02-19T20:25:57.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.307+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.308793+0000) 2024-02-19T20:25:57.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:57.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:57.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:57.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:57.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:57.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:57.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:57.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:58.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:57.863+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.864835+0000) 2024-02-19T20:25:58.037 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:25:57.867+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 62970 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:25:58.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:57.753+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.754611+0000) 2024-02-19T20:25:58.038 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:57.864+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.865745+0000) 2024-02-19T20:25:58.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.864+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:57.865214+0000) 2024-02-19T20:25:58.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:58.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:58.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.996+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:58.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:57.996+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:58.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:58.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:58.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:58.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:58.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:58.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:58.304 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:58.046+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:58.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:58.499+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.500699+0000) 2024-02-19T20:25:58.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:58.307+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.308995+0000) 2024-02-19T20:25:58.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:58.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:58.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:58.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:58.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:58.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:58.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:58.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:59.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:58.753+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:58.754816+0000) 2024-02-19T20:25:59.002 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:58.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:59.002 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:58.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:59.003 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:58.992+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:25:59.003 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:58.993+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:59.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:59.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:59.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:59.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:59.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:59.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:25:59.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:59.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:25:59.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:59.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:59.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:59.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:59.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:59.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:25:59.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:59.449+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:25:59.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:25:59.501+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.502978+0000) 2024-02-19T20:25:59.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:25:59.311+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.312516+0000) 2024-02-19T20:26:00.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:25:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:25:59.756+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:25:59.757085+0000) 2024-02-19T20:26:00.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:00.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:00.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:00.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:00.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:00.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:00.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:00.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:00.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:00.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:00.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:00.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:00.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:00.031+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:00.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:00.031+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:00.716 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:00.315+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.316066+0000) 2024-02-19T20:26:00.716 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:00.504+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.505270+0000) 2024-02-19T20:26:00.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:00.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:00.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:00.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:00.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:00.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:00.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:00.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:00.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:00.758+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:00.759281+0000) 2024-02-19T20:26:01.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:01.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:01.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:01.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:01.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:01.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:01.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:01.014+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:01.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:01.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:01.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:01.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:01.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:01.019+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:01.288 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:01.020+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:01.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:01.318+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.319533+0000) 2024-02-19T20:26:01.756 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:01.506+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.507378+0000) 2024-02-19T20:26:01.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:01.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:01.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:01.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:01.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:01.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:01.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:01.443+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:02.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:01.760+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:01.761338+0000) 2024-02-19T20:26:02.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:02.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:02.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:02.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.017+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:02.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:02.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:02.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:02.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:02.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:02.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:02.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:02.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:02.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:02.509+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.510026+0000) 2024-02-19T20:26:02.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.321+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.322132+0000) 2024-02-19T20:26:02.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:02.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:02.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:02.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:02.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:02.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:02.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:02.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:03.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:02.870+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.871911+0000) 2024-02-19T20:26:03.038 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:02.874+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63009 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:03.039 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:02.762+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.763907+0000) 2024-02-19T20:26:03.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:02.871+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.872902+0000) 2024-02-19T20:26:03.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.870+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:02.871976+0000) 2024-02-19T20:26:03.040 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:03.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:03.041 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.983+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:03.042 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:02.984+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:03.319 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:03.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:03.320 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:03.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:03.321 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:03.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:03.321 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:03.049+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:03.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:03.321+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.322328+0000) 2024-02-19T20:26:03.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:03.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:03.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:03.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:03.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:03.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:03.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:03.506+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:03.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:03.509+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.510214+0000) 2024-02-19T20:26:04.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:03.763+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:03.764131+0000) 2024-02-19T20:26:04.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:03.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:04.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:03.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:04.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:03.939+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:04.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:03.939+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:04.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:04.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:04.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:04.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:04.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:04.024+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:04.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:04.025+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:04.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:04.512+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.513676+0000) 2024-02-19T20:26:04.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:04.324+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.325153+0000) 2024-02-19T20:26:04.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:04.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:04.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:04.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:04.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:04.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:04.762 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:04.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:05.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:04.765+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:04.766327+0000) 2024-02-19T20:26:05.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:04.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:05.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:04.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:05.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:04.987+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:05.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:04.987+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:05.322 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:05.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:05.322 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:05.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:05.322 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:05.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:05.322 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:05.055+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:05.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:05.326+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.327363+0000) 2024-02-19T20:26:05.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:05.514+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.515789+0000) 2024-02-19T20:26:05.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:05.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:05.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:05.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:05.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:05.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:05.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:05.459+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:05.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:05.768+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:05.769822+0000) 2024-02-19T20:26:06.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:05.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:06.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:05.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:06.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:05.944+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:06.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:05.945+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:06.324 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:06.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:06.324 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:06.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:06.324 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:06.044+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:06.324 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:06.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:06.768 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:06.517+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.518304+0000) 2024-02-19T20:26:06.768 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:06.328+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.329422+0000) 2024-02-19T20:26:06.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:06.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:06.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:06.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:06.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:06.497+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:06.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:06.497+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:07.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:06.772+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:06.773357+0000) 2024-02-19T20:26:07.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:06.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:07.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:06.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:07.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:06.951+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:07.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:06.952+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:07.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:07.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:07.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:07.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.011+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:07.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.518+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.519858+0000) 2024-02-19T20:26:07.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:07.331+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.332292+0000) 2024-02-19T20:26:07.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:07.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:07.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:07.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:07.771 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:07.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:07.771 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:07.521+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:08.036 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:07.881+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63049 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:08.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:07.774+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.775081+0000) 2024-02-19T20:26:08.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:07.878+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.879875+0000) 2024-02-19T20:26:08.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:07.879+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.880180+0000) 2024-02-19T20:26:08.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:07.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:08.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:07.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:08.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:07.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:08.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:07.913+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:08.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.878+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:07.879136+0000) 2024-02-19T20:26:08.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:08.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:08.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:08.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:07.974+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:08.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:08.331+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.332452+0000) 2024-02-19T20:26:08.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:08.519+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.520077+0000) 2024-02-19T20:26:08.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:08.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:08.771 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:08.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:08.771 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:08.569+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:08.771 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:08.569+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:09.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:08.774+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:08.775266+0000) 2024-02-19T20:26:09.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:08.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:09.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:08.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:09.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:08.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:09.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:08.980+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:09.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:08.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:09.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:08.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:09.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:08.950+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:09.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:08.950+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:09.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:09.522+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.523633+0000) 2024-02-19T20:26:09.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:09.333+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.334480+0000) 2024-02-19T20:26:09.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:09.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:09.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:09.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:09.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:09.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:09.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:09.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:10.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:09.777+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:09.778839+0000) 2024-02-19T20:26:10.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:09.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:10.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:09.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:10.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:09.903+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:10.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:09.903+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:10.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:10.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:10.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:10.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:10.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:10.014+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:10.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:10.015+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:10.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:10.526+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.527252+0000) 2024-02-19T20:26:10.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:10.335+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.336679+0000) 2024-02-19T20:26:10.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:10.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:10.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:10.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:10.501+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:11.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:10.779+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:10.780967+0000) 2024-02-19T20:26:11.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:10.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:11.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:10.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:11.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:10.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:11.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:10.948+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:11.333 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:11.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:11.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:11.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:11.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:11.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:11.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:11.052+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:11.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:11.528+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.529268+0000) 2024-02-19T20:26:11.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:11.337+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.338919+0000) 2024-02-19T20:26:11.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:11.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:11.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:11.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:11.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:11.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:11.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:11.483+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:12.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:11.782+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:11.783028+0000) 2024-02-19T20:26:12.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:11.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:12.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:11.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:12.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:11.957+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:12.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:11.957+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:12.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:12.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:12.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:12.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:12.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:12.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:12.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:12.084+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:12.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:12.530+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.531950+0000) 2024-02-19T20:26:12.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:12.339+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.340698+0000) 2024-02-19T20:26:12.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:12.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:12.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:12.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:12.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:12.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:12.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:12.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:13.036 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:12.889+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63088 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:13.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:12.783+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.784579+0000) 2024-02-19T20:26:13.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:12.886+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.887119+0000) 2024-02-19T20:26:13.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:12.886+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.887425+0000) 2024-02-19T20:26:13.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:12.886+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:12.887290+0000) 2024-02-19T20:26:13.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:12.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:13.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:12.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:13.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:12.948+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:13.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:12.949+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:13.335 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:13.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:13.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:13.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:13.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:13.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:13.336 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:13.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:13.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:13.339+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.341005+0000) 2024-02-19T20:26:13.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:13.531+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.532196+0000) 2024-02-19T20:26:13.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:13.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:13.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:13.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:13.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:13.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:13.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:13.486+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:14.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:13.783+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:13.784819+0000) 2024-02-19T20:26:14.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:13.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:14.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:13.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:14.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:13.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:14.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:13.975+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:14.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:14.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:14.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:14.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:14.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:14.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:14.339 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:14.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:14.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:14.342+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.343536+0000) 2024-02-19T20:26:14.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:14.533+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.534660+0000) 2024-02-19T20:26:14.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:14.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:14.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:14.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:14.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:14.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:14.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:14.504+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:15.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:14.785+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:14.786881+0000) 2024-02-19T20:26:15.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:14.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:15.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:14.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:15.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:14.993+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:15.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:14.993+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:15.340 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:15.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:15.340 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:15.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:15.340 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:15.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:15.341 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:15.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:15.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:15.344+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.345774+0000) 2024-02-19T20:26:15.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:15.536+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.537190+0000) 2024-02-19T20:26:15.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:15.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:15.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:15.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:15.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:15.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:15.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:15.549+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:16.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:15.788+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:15.789719+0000) 2024-02-19T20:26:16.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:16.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:16.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:16.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:16.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:16.026+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:16.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:16.026+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:16.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:16.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:16.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:16.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:16.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:16.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:16.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:16.071+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:16.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:16.347+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.348162+0000) 2024-02-19T20:26:16.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:16.539+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.540890+0000) 2024-02-19T20:26:16.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:16.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:16.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:16.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:16.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:16.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:16.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:16.504+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:17.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:16.790+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:16.791819+0000) 2024-02-19T20:26:17.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:17.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:17.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:17.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:17.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:17.049+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:17.345 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:17.050+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:17.345 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:17.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:17.345 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:17.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:17.345 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:17.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:17.345 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:17.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:17.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:17.348+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.349783+0000) 2024-02-19T20:26:17.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:17.541+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.542941+0000) 2024-02-19T20:26:17.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:17.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:17.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:17.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:17.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:17.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:17.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:17.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:18.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:17.893+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.894989+0000) 2024-02-19T20:26:18.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:17.792+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.793412+0000) 2024-02-19T20:26:18.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:17.893+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.894571+0000) 2024-02-19T20:26:18.025 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:17.896+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63128 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:18.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:17.892+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:17.893753+0000) 2024-02-19T20:26:18.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:18.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:18.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:18.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:18.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:18.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:18.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:18.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:18.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:18.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:18.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:18.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:18.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:18.082+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:18.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:18.082+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:18.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:18.542+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.543138+0000) 2024-02-19T20:26:18.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:18.349+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.350052+0000) 2024-02-19T20:26:18.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:18.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:18.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:18.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:18.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:18.527+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:18.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:18.528+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:19.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:18.792+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:18.793589+0000) 2024-02-19T20:26:19.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:19.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:19.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:19.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:19.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:19.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:19.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:19.130+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:19.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:19.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:19.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:19.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:19.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:19.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:19.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:19.033+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:19.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:19.544+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.545760+0000) 2024-02-19T20:26:19.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:19.351+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.352235+0000) 2024-02-19T20:26:19.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:19.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:19.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:19.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:19.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:19.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:19.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:19.500+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:20.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:19.794+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:19.795836+0000) 2024-02-19T20:26:20.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:20.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:20.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:20.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:20.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:20.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:20.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:20.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:20.348 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:20.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:20.349 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:20.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:20.349 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:20.104+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:20.349 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:20.104+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:20.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:20.353+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.354195+0000) 2024-02-19T20:26:20.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:20.547+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.548284+0000) 2024-02-19T20:26:20.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:20.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:20.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:20.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:20.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:20.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:20.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:20.489+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:21.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:20.797+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:20.798361+0000) 2024-02-19T20:26:21.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:21.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:21.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:21.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:21.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:21.080+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:21.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:21.080+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:21.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:21.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:21.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:21.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:21.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:21.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:21.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:21.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:21.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:21.355+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.356928+0000) 2024-02-19T20:26:21.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:21.549+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.550360+0000) 2024-02-19T20:26:21.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:21.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:21.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:21.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:21.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:21.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:21.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:21.459+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:22.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:21.799+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:21.800409+0000) 2024-02-19T20:26:22.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:22.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:22.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:22.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:22.353 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:22.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:22.353 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:22.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:22.353 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:22.065+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:22.354 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:22.066+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:22.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:22.357+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.358896+0000) 2024-02-19T20:26:22.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.551+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.552328+0000) 2024-02-19T20:26:22.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:22.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:22.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:22.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:22.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:22.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:22.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:22.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:23.036 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:22.903+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63167 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:23.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:22.800+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.801943+0000) 2024-02-19T20:26:23.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:22.900+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.901539+0000) 2024-02-19T20:26:23.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:22.900+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.901573+0000) 2024-02-19T20:26:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.900+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:22.901439+0000) 2024-02-19T20:26:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.979+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:22.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:23.353 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:23.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:23.353 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:23.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:23.354 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:23.096+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:23.354 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:23.096+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:23.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:23.358+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.359070+0000) 2024-02-19T20:26:23.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:23.551+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.552542+0000) 2024-02-19T20:26:23.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:23.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:23.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:23.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:23.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:23.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:23.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:23.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:24.040 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:23.801+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:23.802149+0000) 2024-02-19T20:26:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:24.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:24.041 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:24.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:24.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:24.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:24.042 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:24.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:24.355 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:24.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:24.356 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:24.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:24.356 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:24.103+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:24.356 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:24.104+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:24.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:24.553+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.554527+0000) 2024-02-19T20:26:24.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:24.360+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.361297+0000) 2024-02-19T20:26:24.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:24.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:24.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:24.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:24.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:24.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:24.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:24.542+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:25.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:24.804+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:24.805708+0000) 2024-02-19T20:26:25.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:25.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:25.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:25.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:25.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:25.151+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:25.287 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:25.151+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:25.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:25.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:25.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.032+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:25.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.033+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:25.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.556+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.557052+0000) 2024-02-19T20:26:25.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:25.364+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.364496+0000) 2024-02-19T20:26:25.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:25.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:25.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:25.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:25.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:25.551+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:25.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:25.551+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:26.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:25.807+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:25.808324+0000) 2024-02-19T20:26:26.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:26.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:26.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:26.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:25.992+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:26.360 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:26.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:26.361 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:26.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:26.361 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:26.106+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:26.361 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:26.107+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:26.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:26.559+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.559620+0000) 2024-02-19T20:26:26.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:26.366+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.366517+0000) 2024-02-19T20:26:26.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:26.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:26.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:26.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:26.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:26.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:26.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:26.552+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:27.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:26.809+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:26.810415+0000) 2024-02-19T20:26:27.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:27.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:27.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:27.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:27.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:27.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:27.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:27.016+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:27.362 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:27.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:27.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:27.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:27.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:27.129+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:27.363 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:27.129+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:27.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:27.368+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.368191+0000) 2024-02-19T20:26:27.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:27.561+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.561308+0000) 2024-02-19T20:26:27.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:27.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:27.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:27.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:27.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:27.510+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:27.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:27.511+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:28.026 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:27.913+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63207 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:28.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:27.909+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.909598+0000) 2024-02-19T20:26:28.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:27.909+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.909257+0000) 2024-02-19T20:26:28.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:27.812+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.812043+0000) 2024-02-19T20:26:28.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:27.910+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:27.910043+0000) 2024-02-19T20:26:28.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:28.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:28.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:28.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:28.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:28.160+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:28.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:28.160+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:28.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:28.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:28.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:28.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:28.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:28.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:28.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:28.057+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:28.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:28.560+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.561505+0000) 2024-02-19T20:26:28.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:28.367+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.368379+0000) 2024-02-19T20:26:28.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:28.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:28.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:28.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:28.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:28.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:28.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:28.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:29.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:28.811+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:28.812221+0000) 2024-02-19T20:26:29.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:29.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:29.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:29.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:29.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:29.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:29.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:29.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:29.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:29.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:29.366 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:29.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:29.366 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:29.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:29.366 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:29.179+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:29.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:29.563+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.564155+0000) 2024-02-19T20:26:29.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:29.369+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.370885+0000) 2024-02-19T20:26:29.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:29.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:29.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:29.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:29.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:29.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:29.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:29.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:30.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:29.813+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:29.814148+0000) 2024-02-19T20:26:30.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:30.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:30.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:30.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:30.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:30.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:30.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:30.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:30.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:30.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:30.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:30.215+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:30.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.565+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.566616+0000) 2024-02-19T20:26:30.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:30.373+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.374466+0000) 2024-02-19T20:26:30.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:30.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:30.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:30.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:30.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:30.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:30.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:30.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:31.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:30.815+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:30.816151+0000) 2024-02-19T20:26:31.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:31.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:31.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.989+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:31.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:30.991+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:31.371 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:31.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:31.371 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:31.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:31.371 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:31.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:31.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:31.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:31.372 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:31.375+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.376899+0000) 2024-02-19T20:26:31.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:31.568+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.569268+0000) 2024-02-19T20:26:31.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:31.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:31.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:31.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:31.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:31.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:31.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:31.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:32.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:31.817+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:31.818145+0000) 2024-02-19T20:26:32.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:31.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:32.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:31.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:32.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:31.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:32.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:31.960+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:32.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:32.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:32.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:32.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:32.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:32.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:32.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:32.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:32.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:32.570+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.571173+0000) 2024-02-19T20:26:32.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:32.378+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.379187+0000) 2024-02-19T20:26:32.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:32.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:32.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:32.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:32.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:32.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:32.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:32.445+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:33.033 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:32.919+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63246 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:33.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:32.916+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.917425+0000) 2024-02-19T20:26:33.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:32.818+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.819717+0000) 2024-02-19T20:26:33.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:32.916+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.917771+0000) 2024-02-19T20:26:33.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:32.916+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:32.917168+0000) 2024-02-19T20:26:33.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:33.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:33.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:33.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:33.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:33.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:33.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:33.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:33.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:33.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:33.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:33.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:33.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:33.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:33.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:33.267+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:33.567 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:33.378+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.379377+0000) 2024-02-19T20:26:33.567 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:33.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:33.567 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:33.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:33.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:33.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:33.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:33.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:33.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:33.570+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.571375+0000) 2024-02-19T20:26:34.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:33.818+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:33.819913+0000) 2024-02-19T20:26:34.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:34.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:34.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:34.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:34.554 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:34.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:34.554 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:34.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:34.554 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:34.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:34.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:34.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:34.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:34.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:34.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:34.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:34.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:34.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:34.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:34.298+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:34.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:34.380+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.381384+0000) 2024-02-19T20:26:34.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.572+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.573482+0000) 2024-02-19T20:26:35.075 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:34.821+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:34.822381+0000) 2024-02-19T20:26:35.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:35.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:35.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:35.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:34.999+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:35.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:35.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:35.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:35.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:35.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:35.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:35.371 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:35.274+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:35.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:35.574+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.575620+0000) 2024-02-19T20:26:35.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:35.382+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.383438+0000) 2024-02-19T20:26:35.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:35.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:35.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:35.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:35.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:35.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:35.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:35.375+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:36.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:35.823+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:35.824588+0000) 2024-02-19T20:26:36.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:36.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:36.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:36.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:36.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:36.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:36.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:36.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:36.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:36.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:36.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:36.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:36.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:36.041+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:36.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:36.041+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:36.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:36.386+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.387117+0000) 2024-02-19T20:26:36.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:36.576+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.577655+0000) 2024-02-19T20:26:36.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:36.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:36.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:36.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:36.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:36.400+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:36.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:36.401+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:37.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:36.825+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:36.826883+0000) 2024-02-19T20:26:37.325 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:37.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:37.326 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:37.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:37.326 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:37.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:37.326 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:37.068+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:37.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:37.578+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.579279+0000) 2024-02-19T20:26:37.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:37.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:37.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:37.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:37.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:37.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:37.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:37.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:37.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:37.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:37.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:37.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:37.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:37.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:37.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:37.331+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:37.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:37.388+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.389662+0000) 2024-02-19T20:26:38.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:37.924+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.925200+0000) 2024-02-19T20:26:38.036 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:37.926+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63286 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:38.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:37.923+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.924550+0000) 2024-02-19T20:26:38.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:37.827+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.828520+0000) 2024-02-19T20:26:38.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:37.923+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:37.924787+0000) 2024-02-19T20:26:38.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:38.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:38.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:38.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:38.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:38.061+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:38.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:38.062+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:38.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:38.578+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.579509+0000) 2024-02-19T20:26:38.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:38.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:38.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:38.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:38.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:38.338+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:38.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:38.339+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:38.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:38.388+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.389884+0000) 2024-02-19T20:26:38.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:38.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:38.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:38.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:38.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:38.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:38.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:38.344+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:39.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:38.827+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:38.828730+0000) 2024-02-19T20:26:39.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:39.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:39.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:39.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:39.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:39.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:39.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:39.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:39.554 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:39.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:39.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:39.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:39.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:39.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:39.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:39.296+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:39.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:39.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:39.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:39.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:39.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:39.367+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:39.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:39.368+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:39.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:39.391+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.392409+0000) 2024-02-19T20:26:39.824 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:39.581+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.582914+0000) 2024-02-19T20:26:40.076 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:39.829+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:39.830730+0000) 2024-02-19T20:26:40.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:40.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:40.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:40.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:40.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:40.051+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:40.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:40.052+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:40.332 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:40.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:40.332 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:40.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:40.332 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:40.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:40.333 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:40.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:40.651 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:40.585+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.586399+0000) 2024-02-19T20:26:40.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:40.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:40.651 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:40.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:40.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:40.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:40.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:40.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:40.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:40.393+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.394561+0000) 2024-02-19T20:26:40.971 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:40.831+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:40.833009+0000) 2024-02-19T20:26:41.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:41.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:41.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:41.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:41.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:41.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:41.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:41.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:41.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:41.307+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:41.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:41.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:41.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:41.308+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:41.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:41.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:41.555 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:41.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:41.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:41.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:41.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:41.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:41.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:41.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:41.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:41.397+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.398232+0000) 2024-02-19T20:26:41.829 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:41.588+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.589960+0000) 2024-02-19T20:26:42.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:41.834+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:41.835131+0000) 2024-02-19T20:26:42.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:42.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:42.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:42.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:42.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:42.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:42.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:42.069+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:42.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:42.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:42.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:42.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:42.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:42.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:42.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:42.318+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:42.587 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:42.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:42.587 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:42.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:42.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:42.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:42.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:42.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:42.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:42.400+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.401111+0000) 2024-02-19T20:26:42.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:42.835+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.836843+0000) 2024-02-19T20:26:42.923 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:42.591+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.592681+0000) 2024-02-19T20:26:43.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:42.931+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.932187+0000) 2024-02-19T20:26:43.286 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:42.934+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63325 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:43.287 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:42.931+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.932420+0000) 2024-02-19T20:26:43.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:42.931+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:42.932616+0000) 2024-02-19T20:26:43.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:43.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:43.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:43.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:43.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:43.068+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:43.288 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:43.069+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:43.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:43.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:43.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:43.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:43.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:43.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:43.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:43.305+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:43.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:43.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:43.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:43.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:43.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:43.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:43.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:43.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:43.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:43.400+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.401314+0000) 2024-02-19T20:26:43.831 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:43.591+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.592911+0000) 2024-02-19T20:26:44.083 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:43.836+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:43.837055+0000) 2024-02-19T20:26:44.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:44.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:44.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:44.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:44.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:44.296+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:44.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:44.296+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:44.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:44.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:44.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:44.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:44.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:44.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:44.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:44.088+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:44.399 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:44.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:44.399 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:44.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:44.399 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:44.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:44.399 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:44.309+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:44.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:44.594+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.595428+0000) 2024-02-19T20:26:44.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:44.402+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.403567+0000) 2024-02-19T20:26:45.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:44.838+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:44.839132+0000) 2024-02-19T20:26:45.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:45.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:45.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:45.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:45.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:45.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:45.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:45.133+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:45.591 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:45.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:45.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:45.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:45.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:45.343+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:45.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:45.343+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:45.592 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:45.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:45.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:45.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:45.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:45.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:45.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:45.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:45.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:45.404+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.405914+0000) 2024-02-19T20:26:45.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:45.840+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.841205+0000) 2024-02-19T20:26:45.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:45.596+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:45.597704+0000) 2024-02-19T20:26:46.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:46.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:46.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:46.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:46.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:46.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:46.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:46.138+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:46.595 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:46.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:46.595 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:46.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:46.595 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:46.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:46.596 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:46.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:46.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:46.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:46.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:46.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:46.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:46.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:46.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:46.313+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:46.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:46.408+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.409641+0000) 2024-02-19T20:26:46.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:46.600+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.601262+0000) 2024-02-19T20:26:46.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:46.843+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:46.844765+0000) 2024-02-19T20:26:47.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:47.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:47.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:47.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:47.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:47.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:47.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:47.178+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:47.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:47.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:47.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:47.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:47.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:47.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:47.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:47.334+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:47.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:47.410+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.411232+0000) 2024-02-19T20:26:47.598 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:47.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:47.598 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:47.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:47.598 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:47.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:47.598 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:47.336+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:47.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:47.845+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.846391+0000) 2024-02-19T20:26:47.930 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:47.602+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.603203+0000) 2024-02-19T20:26:48.285 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:47.942+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63366 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:48.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:47.938+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.939971+0000) 2024-02-19T20:26:48.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:47.939+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.940335+0000) 2024-02-19T20:26:48.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:47.938+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:47.939674+0000) 2024-02-19T20:26:48.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:48.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:48.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:48.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:48.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:48.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:48.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:48.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:48.555 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:48.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:48.556 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:48.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:48.556 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:48.291+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:48.556 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:48.292+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:48.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:48.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:48.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:48.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:48.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:48.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:48.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:48.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:48.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:48.410+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.411442+0000) 2024-02-19T20:26:48.840 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:48.602+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.603410+0000) 2024-02-19T20:26:49.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:48.845+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:48.846600+0000) 2024-02-19T20:26:49.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:49.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:49.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:49.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:49.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:49.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:49.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:49.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:49.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:49.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:49.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:49.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:49.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:49.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:49.493 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:49.184+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:49.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:49.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:49.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:49.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:49.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:49.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:49.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:49.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:49.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:49.412+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.413614+0000) 2024-02-19T20:26:49.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:49.604+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.605910+0000) 2024-02-19T20:26:50.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:49.847+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:49.848800+0000) 2024-02-19T20:26:50.320 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:50.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:50.320 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:50.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:50.320 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:50.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:50.321 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:50.217+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:50.602 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:50.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:50.602 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:50.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:50.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:50.322+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:50.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:50.323+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:50.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:50.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:50.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:50.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:50.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:50.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:50.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:50.339+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:50.604 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:50.414+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.415684+0000) 2024-02-19T20:26:50.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:50.850+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.851328+0000) 2024-02-19T20:26:50.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:50.607+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:50.608406+0000) 2024-02-19T20:26:51.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:51.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:51.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:51.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:51.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:51.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:51.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:51.174+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:51.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:51.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:51.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:51.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:51.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:51.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:51.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:51.347+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:51.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:51.416+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.417769+0000) 2024-02-19T20:26:51.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:51.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:51.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:51.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:51.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:51.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:51.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:51.280+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:51.847 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:51.610+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.611208+0000) 2024-02-19T20:26:52.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:51.852+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:51.853414+0000) 2024-02-19T20:26:52.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:52.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:52.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:52.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:52.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:52.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:52.414 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:52.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:52.414 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:52.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:52.414 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:52.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:52.414 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:52.160+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:52.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:52.161+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:52.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:52.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:52.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:52.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:52.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:52.252+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:52.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:52.253+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:52.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:52.611+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.613006+0000) 2024-02-19T20:26:52.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:52.418+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.419499+0000) 2024-02-19T20:26:53.035 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:52.949+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63405 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:53.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:52.946+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.947578+0000) 2024-02-19T20:26:53.037 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:52.945+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.946954+0000) 2024-02-19T20:26:53.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:52.854+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.855201+0000) 2024-02-19T20:26:53.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:52.946+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:52.947563+0000) 2024-02-19T20:26:53.331 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:53.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:53.331 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:53.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:53.331 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:53.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:53.331 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:53.211+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:53.332 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:53.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:53.332 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:53.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:53.332 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:53.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:53.332 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:53.278+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:53.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:53.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:53.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:53.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:53.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:53.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:53.607 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:53.336+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:53.608 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:53.418+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.419722+0000) 2024-02-19T20:26:53.608 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:53.612+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.613191+0000) 2024-02-19T20:26:53.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:53.854+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:53.855378+0000) 2024-02-19T20:26:54.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:54.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:54.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:54.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:54.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:54.210+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:54.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:54.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:54.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:54.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:54.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:54.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:54.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:54.262+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:54.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:54.263+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:54.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:54.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:54.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:54.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:54.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:54.335+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:54.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:54.336+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:54.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:54.420+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.421769+0000) 2024-02-19T20:26:54.851 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:54.614+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.615550+0000) 2024-02-19T20:26:55.176 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:54.856+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:54.857350+0000) 2024-02-19T20:26:55.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:55.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:55.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:55.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:55.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:55.181+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:55.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:55.182+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:55.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:55.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:55.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:55.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:55.315+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:55.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:55.424+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.425374+0000) 2024-02-19T20:26:55.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:55.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:55.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:55.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:55.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:55.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:55.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:55.295+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:55.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:55.616+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.617705+0000) 2024-02-19T20:26:56.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:55.858+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:55.859570+0000) 2024-02-19T20:26:56.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:56.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:56.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:56.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:56.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:56.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:56.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:56.219+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:56.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:56.294+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:56.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:56.294+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:56.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:56.294+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:56.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:56.295+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:56.423 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:56.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:56.423 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:56.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:56.423 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:56.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:56.423 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:56.268+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:56.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:56.618+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.619915+0000) 2024-02-19T20:26:56.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:56.426+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.427533+0000) 2024-02-19T20:26:57.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:56.860+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:56.861531+0000) 2024-02-19T20:26:57.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:57.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:57.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:57.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:57.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:57.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:57.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:57.254+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:57.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:57.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:57.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:57.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:57.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:57.283+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:57.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:57.284+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:57.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:57.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:57.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:57.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:57.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:57.198+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:57.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:57.199+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:57.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:57.429+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.430361+0000) 2024-02-19T20:26:57.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:57.620+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.621564+0000) 2024-02-19T20:26:58.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:57.954+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.955202+0000) 2024-02-19T20:26:58.036 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:26:57.956+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63446 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:26:58.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:57.953+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.954501+0000) 2024-02-19T20:26:58.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:57.862+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.863193+0000) 2024-02-19T20:26:58.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:57.953+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:57.954601+0000) 2024-02-19T20:26:58.424 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:58.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:58.424 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:58.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:58.424 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:58.246+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:58.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:58.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:58.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:58.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:58.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:58.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:58.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:58.271+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:58.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:58.272+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:58.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:58.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:58.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:58.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:58.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:58.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:58.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:58.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:58.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:58.620+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.621759+0000) 2024-02-19T20:26:58.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:58.429+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.430559+0000) 2024-02-19T20:26:59.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:58.862+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:58.863361+0000) 2024-02-19T20:26:59.299 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:59.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:59.299 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:59.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:59.299 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:59.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:26:59.299 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:59.255+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:59.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:59.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:59.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:59.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:59.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:59.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:26:59.300 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:59.244+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:26:59.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:59.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:59.556 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:59.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:59.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:59.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:26:59.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:59.304+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:26:59.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:26:59.432+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.433033+0000) 2024-02-19T20:26:59.859 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:26:59.622+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.623882+0000) 2024-02-19T20:27:00.200 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:26:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:26:59.864+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:26:59.865334+0000) 2024-02-19T20:27:00.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:00.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:00.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:00.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:00.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:00.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:00.507 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:00.217+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:00.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:00.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:00.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:00.205+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:00.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:00.206+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:00.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:00.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:00.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:00.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:00.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:00.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:00.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:00.263+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:00.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:00.434+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.435647+0000) 2024-02-19T20:27:00.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:00.625+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.626386+0000) 2024-02-19T20:27:01.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:00.866+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:00.867944+0000) 2024-02-19T20:27:01.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:01.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:01.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:01.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:01.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:01.234+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:01.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:01.235+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:01.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:01.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:01.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:01.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:01.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:01.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:01.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:01.215+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:01.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:01.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:01.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:01.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:01.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:01.182+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:01.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:01.183+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:01.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:01.436+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.437754+0000) 2024-02-19T20:27:01.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:01.627+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.628627+0000) 2024-02-19T20:27:02.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:01.869+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:01.870032+0000) 2024-02-19T20:27:02.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:02.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:02.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:02.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:02.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:02.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:02.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:02.230+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:02.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:02.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:02.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:02.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:02.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:02.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:02.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:02.170+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:02.348 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:02.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:02.348 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:02.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:02.349 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:02.188+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:02.349 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:02.188+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:02.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:02.438+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.439494+0000) 2024-02-19T20:27:02.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:02.870+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.871798+0000) 2024-02-19T20:27:02.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:02.629+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.630296+0000) 2024-02-19T20:27:03.225 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:02.965+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63485 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:03.225 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:02.961+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.962426+0000) 2024-02-19T20:27:03.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:03.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:03.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:03.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:03.226 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:03.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:03.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:02.961+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.962387+0000) 2024-02-19T20:27:03.226 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:03.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:03.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:03.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:03.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:03.199+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:03.227 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:03.200+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:03.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:02.960+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:02.961868+0000) 2024-02-19T20:27:03.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:03.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:03.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:03.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:03.227 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:03.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:03.228 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:03.124+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:03.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:03.231+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:03.507 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:03.438+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.439740+0000) 2024-02-19T20:27:03.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:03.629+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.630603+0000) 2024-02-19T20:27:04.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:03.871+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:03.872014+0000) 2024-02-19T20:27:04.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:04.228+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:04.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:04.228+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:04.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:04.228+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:04.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:04.229+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:04.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:04.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:04.437 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:04.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:04.437 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:04.224+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:04.437 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:04.225+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:04.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:04.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:04.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:04.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:04.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:04.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:04.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:04.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:04.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:04.631+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.632797+0000) 2024-02-19T20:27:04.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:04.440+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.441973+0000) 2024-02-19T20:27:05.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:04.873+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:04.874210+0000) 2024-02-19T20:27:05.438 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:05.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:05.438 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:05.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:05.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:05.251+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:05.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:05.251+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:05.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:05.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:05.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:05.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:05.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:05.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:05.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:05.240+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:05.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:05.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:05.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:05.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:05.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:05.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:05.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:05.056+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:05.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:05.443+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.444787+0000) 2024-02-19T20:27:05.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:05.634+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.635042+0000) 2024-02-19T20:27:06.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:05.875+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:05.876350+0000) 2024-02-19T20:27:06.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:06.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:06.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.019+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:06.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.020+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:06.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:06.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:06.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:06.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:06.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:06.216+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:06.442 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:06.217+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:06.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:06.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:06.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:06.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:06.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:06.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:06.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:06.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:06.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.636+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.637613+0000) 2024-02-19T20:27:06.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:06.446+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.447059+0000) 2024-02-19T20:27:07.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:06.879+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:06.880102+0000) 2024-02-19T20:27:07.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:07.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:07.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:07.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:06.979+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:07.442 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:07.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:07.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:07.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:07.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:07.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:07.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:07.250+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:07.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:07.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:07.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:07.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:07.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:07.209+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:07.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:07.210+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:07.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:07.448+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.449235+0000) 2024-02-19T20:27:07.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:07.638+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.639341+0000) 2024-02-19T20:27:08.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:07.970+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.971273+0000) 2024-02-19T20:27:08.036 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:07.972+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63525 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:08.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:07.881+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.882098+0000) 2024-02-19T20:27:08.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:07.969+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.970585+0000) 2024-02-19T20:27:08.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:07.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:08.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:07.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:08.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:07.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:08.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:07.955+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:08.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:07.969+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:07.970475+0000) 2024-02-19T20:27:08.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:08.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:08.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:08.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:08.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:08.196+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:08.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:08.196+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:08.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:08.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:08.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:08.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:08.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:08.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:08.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:08.289+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:08.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:08.638+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.639519+0000) 2024-02-19T20:27:08.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:08.448+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.449461+0000) 2024-02-19T20:27:09.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:08.881+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:08.882336+0000) 2024-02-19T20:27:09.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:08.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:09.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:08.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:09.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:08.969+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:09.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:08.970+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:09.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:09.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:09.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:09.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:09.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:09.306+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:09.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:09.307+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:09.446 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:09.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:09.446 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:09.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:09.446 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:09.225+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:09.446 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:09.225+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:09.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:09.451+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.452070+0000) 2024-02-19T20:27:09.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:09.640+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.641544+0000) 2024-02-19T20:27:10.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:09.884+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:09.885910+0000) 2024-02-19T20:27:10.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:09.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:10.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:09.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:10.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:09.921+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:10.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:09.922+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:10.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:10.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:10.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:10.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:10.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:10.288+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:10.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:10.289+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:10.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:10.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:10.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:10.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:10.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:10.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:10.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:10.259+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:10.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:10.642+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.643704+0000) 2024-02-19T20:27:10.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:10.453+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.454573+0000) 2024-02-19T20:27:11.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:10.887+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:10.888067+0000) 2024-02-19T20:27:11.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:10.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:11.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:10.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:11.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:10.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:11.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:10.967+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:11.319 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:11.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:11.319 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:11.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:11.319 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:11.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:11.320 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:11.299+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:11.639 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:11.455+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.456830+0000) 2024-02-19T20:27:11.639 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:11.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:11.640 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:11.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:11.640 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:11.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:11.640 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:11.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:11.922 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:11.644+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.645840+0000) 2024-02-19T20:27:11.923 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:11.889+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:11.890217+0000) 2024-02-19T20:27:12.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:11.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:12.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:11.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:12.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:11.927+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:12.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:11.928+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:12.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:12.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:12.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:12.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:12.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:12.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:12.642 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:12.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:12.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:12.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:12.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:12.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:12.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:12.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:12.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:12.341+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:12.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:12.457+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.458699+0000) 2024-02-19T20:27:12.924 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:12.890+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.891892+0000) 2024-02-19T20:27:12.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:12.646+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.647626+0000) 2024-02-19T20:27:12.924 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:12.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:12.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:12.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:12.925 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:12.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:13.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:12.976+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.977821+0000) 2024-02-19T20:27:13.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:12.976+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.977575+0000) 2024-02-19T20:27:13.286 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:12.980+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63564 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:13.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:12.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:13.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:12.977+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:12.978237+0000) 2024-02-19T20:27:13.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:13.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:13.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:13.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:13.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:13.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:13.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:13.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:13.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:13.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:13.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:13.311+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:13.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:13.311+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:13.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:13.311+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:13.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:13.457+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.458904+0000) 2024-02-19T20:27:13.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:13.646+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.647849+0000) 2024-02-19T20:27:14.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:13.892+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:13.892123+0000) 2024-02-19T20:27:14.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:13.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:14.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:13.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:14.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:13.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:14.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:13.960+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:14.644 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:14.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:14.644 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:14.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:14.644 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:14.395+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:14.644 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:14.396+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:14.645 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:14.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:14.645 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:14.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:14.645 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:14.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:14.645 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:14.348+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:14.645 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:14.460+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.460983+0000) 2024-02-19T20:27:14.908 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:14.894+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.894260+0000) 2024-02-19T20:27:14.909 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:14.650+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:14.650405+0000) 2024-02-19T20:27:15.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:14.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:15.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:14.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:15.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:14.915+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:15.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:14.916+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:15.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:15.409+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:15.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:15.409+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:15.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:15.409+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:15.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:15.410+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:15.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:15.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:15.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:15.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:15.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:15.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:15.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:15.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:15.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:15.462+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.463764+0000) 2024-02-19T20:27:15.947 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:15.895+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.896317+0000) 2024-02-19T20:27:15.947 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:15.652+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:15.653154+0000) 2024-02-19T20:27:16.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:15.952+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:16.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:15.952+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:16.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:15.952+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:16.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:15.953+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:16.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:16.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:16.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:16.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:16.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:16.402+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:16.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:16.402+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:16.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:16.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:16.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:16.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:16.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:16.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:16.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:16.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:16.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:16.464+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.465842+0000) 2024-02-19T20:27:16.891 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:16.655+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.656040+0000) 2024-02-19T20:27:17.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:16.897+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:16.898457+0000) 2024-02-19T20:27:17.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:17.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:17.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:17.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:17.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:17.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:17.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:17.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:17.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:17.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:17.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:17.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:17.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:17.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:17.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:17.452+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:17.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:17.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:17.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:17.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:17.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:17.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:17.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:17.305+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:17.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:17.466+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.467582+0000) 2024-02-19T20:27:17.893 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:17.657+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.658172+0000) 2024-02-19T20:27:18.285 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:17.988+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63603 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:18.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:17.984+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.985848+0000) 2024-02-19T20:27:18.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:17.899+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.900353+0000) 2024-02-19T20:27:18.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:17.984+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.985376+0000) 2024-02-19T20:27:18.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:17.984+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:17.985218+0000) 2024-02-19T20:27:18.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:18.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:18.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:18.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:18.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:18.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:18.287 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:18.024+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:18.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:18.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:18.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:18.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:18.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:18.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:18.652 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:18.480+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:18.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:18.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:18.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:18.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:18.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:18.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:18.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:18.320+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:18.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:18.466+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.467778+0000) 2024-02-19T20:27:18.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:18.899+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.900593+0000) 2024-02-19T20:27:18.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:18.657+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:18.658397+0000) 2024-02-19T20:27:19.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:19.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:19.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:19.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:19.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:19.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:19.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:19.051+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:19.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:19.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:19.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:19.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:19.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:19.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:19.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:19.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:19.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:19.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:19.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:19.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:19.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:19.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:19.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:19.278+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:19.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:19.468+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.469829+0000) 2024-02-19T20:27:19.896 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:19.659+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.660785+0000) 2024-02-19T20:27:20.243 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:19.901+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:19.902756+0000) 2024-02-19T20:27:20.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:20.243 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:20.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:20.244 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.022+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:20.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:20.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:20.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:20.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:20.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:20.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:20.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:20.466+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:20.510 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:20.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:20.510 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:20.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:20.510 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:20.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:20.510 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:20.299+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:20.510 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:20.470+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.471921+0000) 2024-02-19T20:27:20.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.662+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.663125+0000) 2024-02-19T20:27:21.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:20.904+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:20.905242+0000) 2024-02-19T20:27:21.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:21.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:21.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.973+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:21.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:20.974+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:21.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:21.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:21.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:21.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:21.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:21.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:21.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:21.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:21.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:21.473+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.474036+0000) 2024-02-19T20:27:21.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:21.664+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.665191+0000) 2024-02-19T20:27:21.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:21.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:21.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:21.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:21.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:21.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:21.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:21.448+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:22.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:21.906+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:21.907285+0000) 2024-02-19T20:27:22.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:21.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:22.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:21.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:22.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:21.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:22.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:21.936+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:22.440 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:22.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:22.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:22.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:22.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:22.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:22.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:22.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:22.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:22.476+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.477075+0000) 2024-02-19T20:27:22.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:22.665+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.666921+0000) 2024-02-19T20:27:22.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:22.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:22.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:22.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:22.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:22.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:22.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:22.447+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:23.035 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:22.995+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63639 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:23.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:22.993+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.994140+0000) 2024-02-19T20:27:23.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:22.907+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.908950+0000) 2024-02-19T20:27:23.037 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:22.992+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.993723+0000) 2024-02-19T20:27:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:22.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:22.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:22.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:23.038 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:22.936+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:23.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:22.992+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:22.993805+0000) 2024-02-19T20:27:23.440 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:23.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:23.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:23.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:23.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:23.288+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:23.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:23.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:23.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:23.666+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.667158+0000) 2024-02-19T20:27:23.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:23.476+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.477282+0000) 2024-02-19T20:27:23.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:23.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:23.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:23.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:23.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:23.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:23.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:23.447+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:24.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:23.908+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:23.909128+0000) 2024-02-19T20:27:24.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:23.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:24.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:23.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:24.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:23.965+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:24.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:23.966+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:24.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:24.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:24.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:24.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:24.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:24.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:24.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:24.258+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:24.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:24.478+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.479329+0000) 2024-02-19T20:27:24.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:24.668+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.669249+0000) 2024-02-19T20:27:24.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:24.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:24.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:24.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:24.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:24.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:24.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:24.425+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:25.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:24.910+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:24.911504+0000) 2024-02-19T20:27:25.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:24.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:25.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:24.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:25.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:24.924+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:25.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:24.925+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:25.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:25.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:25.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:25.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:25.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:25.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:25.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:25.267+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:25.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:25.670+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.671736+0000) 2024-02-19T20:27:25.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:25.480+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.481780+0000) 2024-02-19T20:27:25.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:25.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:25.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:25.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:25.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:25.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:25.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:25.423+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:26.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:25.912+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:25.913900+0000) 2024-02-19T20:27:26.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:25.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:26.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:25.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:26.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:25.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:26.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:25.900+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:26.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:26.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:26.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:26.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:26.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:26.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:26.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:26.242+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:26.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:26.672+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.673813+0000) 2024-02-19T20:27:26.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:26.484+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.485472+0000) 2024-02-19T20:27:26.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:26.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:26.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:26.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:26.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:26.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:26.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:26.445+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:27.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:26.916+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:26.917284+0000) 2024-02-19T20:27:27.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:26.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:27.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:26.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:27.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:26.896+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:27.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:26.897+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:27.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:27.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:27.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:27.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:27.451 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:27.195+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:27.451 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:27.196+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:27.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:27.675+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.676798+0000) 2024-02-19T20:27:27.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:27.487+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.488528+0000) 2024-02-19T20:27:27.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:27.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:27.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:27.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:27.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:27.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:27.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:27.456+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:28.035 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:28.003+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63678 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:28.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:28.000+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.001139+0000) 2024-02-19T20:27:28.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:27.919+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:27.920065+0000) 2024-02-19T20:27:28.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:27.999+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.000261+0000) 2024-02-19T20:27:28.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:27.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:28.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:27.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:28.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:27.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:28.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:27.860+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:28.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:28.002+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.003566+0000) 2024-02-19T20:27:28.473 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:28.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:28.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:28.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:28.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:28.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:28.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:28.221+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:28.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:28.487+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.488732+0000) 2024-02-19T20:27:28.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:28.675+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.676987+0000) 2024-02-19T20:27:28.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:28.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:28.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:28.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:28.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:28.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:28.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:28.480+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:29.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:28.919+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:28.920266+0000) 2024-02-19T20:27:29.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:28.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:29.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:28.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:29.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:28.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:29.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:28.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:29.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:29.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:29.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:29.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:29.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:29.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:29.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:29.215+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:29.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:29.491+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.492212+0000) 2024-02-19T20:27:29.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:29.678+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.679039+0000) 2024-02-19T20:27:29.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:29.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:29.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:29.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:29.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:29.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:29.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:29.519+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:30.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:29.922+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:29.923136+0000) 2024-02-19T20:27:30.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:29.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:30.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:29.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:30.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:29.873+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:30.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:29.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:30.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:30.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:30.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:30.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:30.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:30.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:30.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:30.181+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:30.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:30.680+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.681317+0000) 2024-02-19T20:27:30.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:30.493+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.494381+0000) 2024-02-19T20:27:30.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:30.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:30.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:30.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:30.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:30.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:30.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:30.558+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:31.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:30.924+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:30.925837+0000) 2024-02-19T20:27:31.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:30.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:31.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:30.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:31.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:30.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:31.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:30.841+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:31.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:31.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:31.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:31.135+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:31.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:31.136+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:31.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:31.136+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:31.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:31.683+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.684058+0000) 2024-02-19T20:27:31.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:31.496+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.497012+0000) 2024-02-19T20:27:31.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:31.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:31.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:31.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:31.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:31.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:31.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:31.570+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:32.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:31.926+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:31.927858+0000) 2024-02-19T20:27:32.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:31.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:32.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:31.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:32.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:31.884+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:32.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:31.885+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:32.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:32.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:32.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:32.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:32.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:32.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:32.494 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:32.181+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:32.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:32.685+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.686174+0000) 2024-02-19T20:27:32.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:32.498+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.499081+0000) 2024-02-19T20:27:32.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:32.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:32.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:32.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:32.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:32.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:32.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:32.555+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:33.035 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:33.011+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63718 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:33.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:33.008+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.009499+0000) 2024-02-19T20:27:33.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:32.928+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:32.929617+0000) 2024-02-19T20:27:33.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:33.008+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.009337+0000) 2024-02-19T20:27:33.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:32.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:33.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:32.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:33.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:32.839+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:33.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:32.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:33.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:33.008+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.009383+0000) 2024-02-19T20:27:33.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:33.133+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:33.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:33.133+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:33.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:33.133+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:33.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:33.133+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:33.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:33.498+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.499302+0000) 2024-02-19T20:27:33.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:33.685+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.686398+0000) 2024-02-19T20:27:33.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:33.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:33.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:33.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:33.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:33.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:33.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:33.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:34.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:33.928+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:33.929825+0000) 2024-02-19T20:27:34.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:33.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:34.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:33.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:34.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:33.864+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:34.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:33.865+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:34.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:34.117+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:34.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:34.117+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:34.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:34.117+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:34.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:34.118+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:34.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:34.687+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.688462+0000) 2024-02-19T20:27:34.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:34.500+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.501716+0000) 2024-02-19T20:27:34.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:34.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:34.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:34.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:34.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:34.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:34.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:34.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:35.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:34.930+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:34.931837+0000) 2024-02-19T20:27:35.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:34.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:35.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:34.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:35.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:34.909+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:35.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:34.910+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:35.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:35.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:35.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:35.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:35.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:35.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:35.497 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:35.071+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:35.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:35.689+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.690693+0000) 2024-02-19T20:27:35.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:35.502+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.503874+0000) 2024-02-19T20:27:35.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:35.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:35.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:35.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:35.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:35.552+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:35.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:35.552+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:36.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:35.932+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:35.933862+0000) 2024-02-19T20:27:36.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:35.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:36.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:35.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:36.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:35.911+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:36.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:35.912+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:36.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:36.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:36.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:36.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:36.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:36.062+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:36.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:36.062+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:36.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:36.691+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.692804+0000) 2024-02-19T20:27:36.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:36.506+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.507573+0000) 2024-02-19T20:27:36.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:36.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:36.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:36.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:36.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:36.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:36.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:36.554+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:37.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:36.936+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:36.937518+0000) 2024-02-19T20:27:37.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:36.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:37.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:36.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:37.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:36.889+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:37.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:36.890+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:37.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:37.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:37.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:37.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:37.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:37.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:37.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:37.084+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:37.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:37.693+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.694556+0000) 2024-02-19T20:27:37.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:37.508+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.509386+0000) 2024-02-19T20:27:37.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:37.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:37.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:37.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:37.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:37.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:37.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:37.573+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:38.034 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:38.018+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63757 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:38.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:38.015+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.016994+0000) 2024-02-19T20:27:38.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:37.938+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:37.939217+0000) 2024-02-19T20:27:38.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:38.015+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.016446+0000) 2024-02-19T20:27:38.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:37.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:38.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:37.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:38.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:37.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:38.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:37.888+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:38.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:38.014+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.015900+0000) 2024-02-19T20:27:38.504 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:38.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:38.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:38.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:38.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:38.126+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:38.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:38.127+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:38.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:38.693+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.694768+0000) 2024-02-19T20:27:38.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:38.508+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.509588+0000) 2024-02-19T20:27:38.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:38.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:38.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:38.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:38.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:38.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:38.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:38.620+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:39.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:38.938+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:38.939402+0000) 2024-02-19T20:27:39.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:38.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:39.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:38.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:39.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:38.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:39.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:38.876+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:39.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:39.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:39.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:39.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:39.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:39.116+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:39.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:39.116+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:39.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:39.695+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.696848+0000) 2024-02-19T20:27:39.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:39.512+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.513144+0000) 2024-02-19T20:27:39.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:39.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:39.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:39.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:39.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:39.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:39.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:39.660+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:40.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:39.941+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:39.942869+0000) 2024-02-19T20:27:40.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:39.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:40.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:39.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:40.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:39.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:40.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:39.832+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:40.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:40.125+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:40.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:40.125+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:40.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:40.125+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:40.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:40.125+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:40.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:40.514+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.515318+0000) 2024-02-19T20:27:40.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:40.697+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.698979+0000) 2024-02-19T20:27:40.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:40.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:40.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:40.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:40.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:40.668+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:40.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:40.669+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:41.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:40.943+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:40.944996+0000) 2024-02-19T20:27:41.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:40.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:41.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:40.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:41.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:40.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:41.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:40.789+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:41.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:41.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:41.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:41.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:41.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:41.132+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:41.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:41.133+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:41.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:41.701+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.702462+0000) 2024-02-19T20:27:41.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:41.517+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.518915+0000) 2024-02-19T20:27:41.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:41.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:41.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:41.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:41.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:41.626+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:41.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:41.627+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:42.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:41.945+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:41.946969+0000) 2024-02-19T20:27:42.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:41.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:42.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:41.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:42.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:41.786+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:42.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:41.787+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:42.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:42.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:42.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:42.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:42.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:42.118+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:42.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:42.119+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:42.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:42.520+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.521065+0000) 2024-02-19T20:27:42.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:42.703+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.704607+0000) 2024-02-19T20:27:42.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:42.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:42.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:42.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:42.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:42.764+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:42.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:42.764+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:42.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:42.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:42.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:42.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:42.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:42.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:42.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:42.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:43.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:43.023+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.024087+0000) 2024-02-19T20:27:43.035 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:43.026+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63797 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:43.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:42.948+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:42.949920+0000) 2024-02-19T20:27:43.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:43.022+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.023612+0000) 2024-02-19T20:27:43.036 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:43.023+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.024686+0000) 2024-02-19T20:27:43.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:43.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:43.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:43.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:43.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:43.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:43.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:43.115+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:43.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:43.703+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.704813+0000) 2024-02-19T20:27:43.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:43.520+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.521254+0000) 2024-02-19T20:27:43.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:43.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:43.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:43.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:43.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:43.624+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:43.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:43.624+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:44.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:43.949+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:43.950130+0000) 2024-02-19T20:27:44.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:43.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:44.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:43.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:44.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:43.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:44.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:43.807+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:44.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:44.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:44.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:44.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:44.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:44.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:44.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:44.154+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:44.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:44.523+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.524636+0000) 2024-02-19T20:27:44.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:44.706+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.707082+0000) 2024-02-19T20:27:44.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:44.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:44.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:44.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:44.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:44.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:44.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:44.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:45.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:44.951+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:44.952107+0000) 2024-02-19T20:27:45.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:44.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:45.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:44.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:45.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:44.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:45.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:44.780+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:45.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:45.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:45.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:45.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:45.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:45.197+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:45.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:45.198+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:45.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:45.708+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.709120+0000) 2024-02-19T20:27:45.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:45.527+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.528074+0000) 2024-02-19T20:27:45.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:45.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:45.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:45.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:45.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:45.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:45.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:45.702+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:46.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:45.953+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:45.954702+0000) 2024-02-19T20:27:46.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:45.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:46.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:45.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:46.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:45.829+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:46.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:45.829+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:46.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:46.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:46.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:46.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:46.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:46.179+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:46.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:46.180+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:46.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:46.530+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.531505+0000) 2024-02-19T20:27:46.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:46.710+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.711568+0000) 2024-02-19T20:27:46.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:46.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:46.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:46.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:46.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:46.709+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:46.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:46.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:47.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:46.955+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:46.956740+0000) 2024-02-19T20:27:47.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:46.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:47.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:46.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:47.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:46.788+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:47.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:46.789+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:47.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:47.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:47.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:47.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:47.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:47.207+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:47.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:47.208+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:47.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:47.533+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.534405+0000) 2024-02-19T20:27:47.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:47.712+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.713840+0000) 2024-02-19T20:27:47.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:47.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:47.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:47.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:47.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:47.674+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:47.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:47.675+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:48.035 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:48.034+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63836 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:48.035 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:48.031+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.032566+0000) 2024-02-19T20:27:48.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:47.957+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:47.958458+0000) 2024-02-19T20:27:48.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:48.030+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.031704+0000) 2024-02-19T20:27:48.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:47.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:48.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:47.792+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:48.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:47.793+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:48.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:47.794+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:48.036 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:48.029+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.030812+0000) 2024-02-19T20:27:48.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:48.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:48.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:48.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:48.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:48.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:48.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:48.174+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:48.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:48.533+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.534620+0000) 2024-02-19T20:27:48.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:48.713+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.714051+0000) 2024-02-19T20:27:48.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:48.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:48.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:48.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:48.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:48.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:48.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:48.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:49.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:48.957+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:48.958649+0000) 2024-02-19T20:27:49.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:48.808+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:49.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:48.808+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:49.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:48.808+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:49.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:48.809+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:49.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:49.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:49.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:49.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:49.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:49.139+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:49.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:49.140+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:49.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:49.537+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.538121+0000) 2024-02-19T20:27:49.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:49.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:49.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:49.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:49.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:49.666+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:49.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:49.667+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:49.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:49.715+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.716068+0000) 2024-02-19T20:27:49.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:49.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:49.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:49.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:49.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:49.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:50.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:49.959+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:49.960622+0000) 2024-02-19T20:27:50.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:49.761+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:50.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:50.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:50.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:50.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:50.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:50.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:50.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:50.139+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:50.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:50.717+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.718726+0000) 2024-02-19T20:27:50.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:50.540+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.541731+0000) 2024-02-19T20:27:50.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:50.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:50.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:50.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:50.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:50.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:50.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:50.629+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:51.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:50.961+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:50.962642+0000) 2024-02-19T20:27:51.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:50.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:51.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:50.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:51.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:50.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:51.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:50.808+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:51.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:51.166+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:51.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:51.166+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:51.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:51.166+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:51.498 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:51.166+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:51.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:51.543+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.544614+0000) 2024-02-19T20:27:51.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:51.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:51.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:51.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:51.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:51.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:51.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:51.582+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:51.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:51.720+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.721320+0000) 2024-02-19T20:27:51.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:51.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:51.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:51.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:51.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:51.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:52.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:51.963+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:51.964804+0000) 2024-02-19T20:27:52.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:51.761+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:52.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:52.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:52.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:52.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:52.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:52.180+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:52.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:52.181+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:52.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:52.545+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.546776+0000) 2024-02-19T20:27:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:52.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:52.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:52.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:52.571+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:52.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:52.722+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.723446+0000) 2024-02-19T20:27:52.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:52.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:52.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:52.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:52.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:52.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:52.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:52.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:53.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:52.965+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:52.966706+0000) 2024-02-19T20:27:53.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:53.038+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.039889+0000) 2024-02-19T20:27:53.285 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:53.042+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63876 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:53.285 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:53.038+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.039956+0000) 2024-02-19T20:27:53.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:53.039+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.040077+0000) 2024-02-19T20:27:53.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:53.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:53.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:53.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:53.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:53.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:53.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:53.209+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:53.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:53.545+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.546990+0000) 2024-02-19T20:27:53.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:53.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:53.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:53.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:53.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:53.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:53.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:53.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:53.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:53.722+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.723650+0000) 2024-02-19T20:27:53.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:53.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:53.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:53.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:53.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:53.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:53.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:53.740+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:54.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:53.965+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:53.966883+0000) 2024-02-19T20:27:54.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:54.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:54.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:54.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:54.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:54.227+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:54.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:54.227+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:54.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:54.548+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.549030+0000) 2024-02-19T20:27:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:54.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:54.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:54.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:54.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:54.568+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:54.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:54.726+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.727213+0000) 2024-02-19T20:27:54.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:54.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:54.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:54.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:54.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:54.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:54.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:54.767+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:55.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:54.969+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:54.970274+0000) 2024-02-19T20:27:55.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:55.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:55.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:55.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:55.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:55.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:55.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:55.218+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:55.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:55.729+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.730829+0000) 2024-02-19T20:27:55.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:55.550+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.551079+0000) 2024-02-19T20:27:55.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:55.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:55.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:55.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:55.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:55.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:55.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:55.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:56.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:55.972+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:55.973548+0000) 2024-02-19T20:27:56.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:55.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:56.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:55.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:56.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:55.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:56.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:55.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:56.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:56.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:56.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:56.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:56.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:56.228+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:56.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:56.228+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:56.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:56.553+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.554775+0000) 2024-02-19T20:27:56.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:56.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:56.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:56.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:56.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:56.584+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:56.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:56.585+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:56.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:56.731+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.732866+0000) 2024-02-19T20:27:56.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:56.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:56.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:56.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:56.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:56.777+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:57.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:56.778+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:57.035 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:56.974+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:56.975605+0000) 2024-02-19T20:27:57.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:57.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:57.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:57.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:57.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:57.232+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:57.513 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:57.233+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:57.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:57.555+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.556584+0000) 2024-02-19T20:27:57.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:57.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:57.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:57.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:57.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:57.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:57.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:57.566+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:57.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:57.733+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.734652+0000) 2024-02-19T20:27:57.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:57.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:57.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:57.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:57.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:57.767+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:57.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:57.768+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:58.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:57.976+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:57.977418+0000) 2024-02-19T20:27:58.512 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:27:58.050+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63915 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:27:58.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:58.046+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.047435+0000) 2024-02-19T20:27:58.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:58.046+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.047684+0000) 2024-02-19T20:27:58.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:58.046+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.047565+0000) 2024-02-19T20:27:58.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:58.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:58.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:58.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:58.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:58.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:58.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:58.209+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:58.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:58.555+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.556797+0000) 2024-02-19T20:27:58.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:58.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:58.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:58.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:58.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:58.529+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:58.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:58.530+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:58.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:58.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:58.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:58.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:58.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:58.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:58.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:58.733+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.734837+0000) 2024-02-19T20:27:58.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:58.734+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:27:59.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:58.976+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:58.977624+0000) 2024-02-19T20:27:59.480 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:59.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:59.480 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:59.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:59.481 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:59.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:27:59.481 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:59.242+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:59.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:27:59.558+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.559279+0000) 2024-02-19T20:27:59.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:59.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:59.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:59.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:59.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:59.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:27:59.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:59.487+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:27:59.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:59.736+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.737319+0000) 2024-02-19T20:27:59.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:59.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:59.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:59.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:27:59.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:59.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:00.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:27:59.978+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:27:59.979626+0000) 2024-02-19T20:28:00.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:27:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:27:59.752+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:00.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:00.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:00.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:00.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:00.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:00.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:00.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:00.245+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:00.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:00.561+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.562579+0000) 2024-02-19T20:28:00.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:00.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:00.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:00.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:00.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:00.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:00.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:00.448+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:00.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:00.740+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.740075+0000) 2024-02-19T20:28:00.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:00.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:00.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:00.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:00.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:00.796+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:00.976 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:00.797+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:01.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:00.983+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:00.983115+0000) 2024-02-19T20:28:01.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:01.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:01.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:01.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:01.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:01.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:01.557 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:01.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:01.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:01.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:01.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:01.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:01.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:01.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:01.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:01.415+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:01.977 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:01.564+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.564612+0000) 2024-02-19T20:28:01.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:01.742+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.742340+0000) 2024-02-19T20:28:01.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:01.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:01.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:01.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:01.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:01.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:01.978 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:01.750+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:02.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:01.984+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:01.985168+0000) 2024-02-19T20:28:02.534 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:02.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:02.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:02.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:02.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:02.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:02.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:02.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:02.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:02.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:02.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:02.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:02.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:02.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:02.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:02.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:02.979 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:02.565+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.566876+0000) 2024-02-19T20:28:02.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:02.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:02.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:02.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:02.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:02.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:02.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:02.700+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:02.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:02.743+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.744285+0000) 2024-02-19T20:28:03.266 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:03.058+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63955 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:03.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:03.054+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.055865+0000) 2024-02-19T20:28:03.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:02.986+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:02.987569+0000) 2024-02-19T20:28:03.267 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:03.055+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.056164+0000) 2024-02-19T20:28:03.267 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:03.054+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.055732+0000) 2024-02-19T20:28:03.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:03.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:03.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:03.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:03.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:03.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:03.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:03.273+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:03.535 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:03.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:03.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:03.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:03.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:03.408+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:03.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:03.408+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:03.979 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:03.566+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.567084+0000) 2024-02-19T20:28:03.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:03.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:03.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:03.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:03.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:03.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:03.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:03.740+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:03.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:03.743+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.744494+0000) 2024-02-19T20:28:04.236 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:03.986+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:03.987769+0000) 2024-02-19T20:28:04.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:04.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:04.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:04.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:04.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:04.382+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:04.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:04.383+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:04.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:04.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:04.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:04.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:04.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:04.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:04.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:04.243+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:04.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:04.568+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.569046+0000) 2024-02-19T20:28:04.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:04.745+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.746501+0000) 2024-02-19T20:28:04.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:04.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:04.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:04.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:04.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:04.779+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:05.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:04.989+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:04.990232+0000) 2024-02-19T20:28:05.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:04.780+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:05.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:05.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:05.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:05.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:05.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:05.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:05.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:05.239+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:05.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:05.570+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.571554+0000) 2024-02-19T20:28:05.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:05.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:05.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:05.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:05.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:05.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:05.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:05.412+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:06.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:05.991+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.992202+0000) 2024-02-19T20:28:06.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:05.747+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:05.748552+0000) 2024-02-19T20:28:06.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:05.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:06.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:05.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:06.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:05.763+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:06.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:05.764+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:06.447 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:06.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:06.447 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:06.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:06.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:06.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:06.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:06.269+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:06.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:06.572+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.573563+0000) 2024-02-19T20:28:06.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:06.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:06.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:06.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:06.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:06.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:06.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:06.454+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:06.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:06.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:06.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:06.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:06.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:06.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:06.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:06.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:06.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:06.751+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.752114+0000) 2024-02-19T20:28:07.231 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:06.993+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:06.994281+0000) 2024-02-19T20:28:07.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:07.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:07.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:07.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:07.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:07.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:07.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:07.437+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:07.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:07.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:07.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:07.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:07.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:07.238+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:07.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:07.238+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:07.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:07.574+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.575462+0000) 2024-02-19T20:28:07.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:07.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:07.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:07.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:07.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:07.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:07.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:07.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:07.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:07.752+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.753988+0000) 2024-02-19T20:28:08.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:07.995+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:07.996037+0000) 2024-02-19T20:28:08.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:08.065+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 63994 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:08.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:08.063+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.064107+0000) 2024-02-19T20:28:08.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:08.062+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.063682+0000) 2024-02-19T20:28:08.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:08.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:08.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:08.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:08.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:08.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:08.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:08.231+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:08.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:08.063+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.064304+0000) 2024-02-19T20:28:08.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:08.574+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.575638+0000) 2024-02-19T20:28:08.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:08.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:08.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:08.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:08.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:08.415+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:08.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:08.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:08.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:08.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:08.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:08.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:08.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:08.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:08.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:08.706+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:08.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:08.753+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.754171+0000) 2024-02-19T20:28:09.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:08.995+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:08.996205+0000) 2024-02-19T20:28:09.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:09.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:09.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:09.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:09.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:09.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:09.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:09.224+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:09.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:09.576+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.577788+0000) 2024-02-19T20:28:09.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:09.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:09.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:09.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:09.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:09.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:09.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:09.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:09.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:09.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:09.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:09.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:09.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:09.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:09.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:09.690+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:10.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:09.756+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.757754+0000) 2024-02-19T20:28:10.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:09.997+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:09.998254+0000) 2024-02-19T20:28:10.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:10.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:10.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:10.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:10.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:10.199+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:10.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:10.200+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:10.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:10.579+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.580345+0000) 2024-02-19T20:28:10.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:10.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:10.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:10.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:10.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:10.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:10.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:10.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:10.979 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:10.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:10.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:10.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:10.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:10.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:10.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:10.699+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:10.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:10.759+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:10.760995+0000) 2024-02-19T20:28:11.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:10.999+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.000752+0000) 2024-02-19T20:28:11.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:11.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:11.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:11.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:11.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:11.194+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:11.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:11.195+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:11.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:11.581+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.582383+0000) 2024-02-19T20:28:11.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:11.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:11.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:11.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:11.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:11.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:11.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:11.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:11.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:11.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:11.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:11.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:11.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:11.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:11.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:11.708+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:11.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:11.763+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:11.764652+0000) 2024-02-19T20:28:12.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:12.001+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.002704+0000) 2024-02-19T20:28:12.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:12.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:12.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:12.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:12.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:12.157+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:12.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:12.157+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:12.715 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:12.583+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.584170+0000) 2024-02-19T20:28:12.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:12.369+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:12.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:12.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:12.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:12.370+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:12.717 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:12.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:12.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:12.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:12.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:12.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:12.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:12.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:12.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:12.723+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:12.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:12.765+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:12.766490+0000) 2024-02-19T20:28:13.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:13.073+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64034 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:13.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:13.070+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.071234+0000) 2024-02-19T20:28:13.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:13.003+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.004900+0000) 2024-02-19T20:28:13.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:13.070+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.071435+0000) 2024-02-19T20:28:13.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:13.070+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.071809+0000) 2024-02-19T20:28:13.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:13.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:13.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:13.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:13.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:13.173+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:13.286 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:13.173+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:13.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:13.583+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.584423+0000) 2024-02-19T20:28:13.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:13.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:13.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:13.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:13.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:13.352+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:41.644421+0000 (oldest deadline 2024-02-19T20:18:01.644421+0000) 2024-02-19T20:28:13.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:13.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:14.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:14.004+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.005065+0000) 2024-02-19T20:28:14.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:13.765+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:13.766689+0000) 2024-02-19T20:28:14.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:13.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:14.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:13.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:14.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:13.771+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:14.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:13.771+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:14.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:14.321+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:14.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:14.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:14.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:14.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:14.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:14.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:14.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:14.217+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:14.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:14.585+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.586398+0000) 2024-02-19T20:28:14.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:14.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:14.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:14.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:14.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:14.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:14.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:14.726+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:14.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:14.769+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:14.770152+0000) 2024-02-19T20:28:15.084 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:15.006+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.007050+0000) 2024-02-19T20:28:15.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:15.339+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:15.514 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:15.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:15.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:15.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:15.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:15.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:15.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:15.185+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:15.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:15.587+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.588646+0000) 2024-02-19T20:28:15.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:15.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:15.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:15.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:15.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:15.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:15.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:15.727+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:15.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:15.771+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:15.772151+0000) 2024-02-19T20:28:16.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:16.007+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.008991+0000) 2024-02-19T20:28:16.339 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:16.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:16.340 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:16.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:16.340 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:16.203+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:16.340 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:16.204+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:16.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:16.346+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:16.691 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:16.590+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.591102+0000) 2024-02-19T20:28:16.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:16.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:16.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:16.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:16.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:16.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:16.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:16.698+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:16.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:16.773+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:16.774355+0000) 2024-02-19T20:28:17.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:17.010+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.011145+0000) 2024-02-19T20:28:17.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:17.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:17.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:17.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:17.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:17.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:17.285 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:17.231+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:17.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:17.318+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:17.980 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:17.591+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.592945+0000) 2024-02-19T20:28:17.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:17.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:17.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:17.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:17.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:17.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:17.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:17.709+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:17.981 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:17.775+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:17.776237+0000) 2024-02-19T20:28:18.243 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:18.080+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64073 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:18.244 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:18.076+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.077441+0000) 2024-02-19T20:28:18.244 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:18.011+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.012902+0000) 2024-02-19T20:28:18.244 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:18.077+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.079003+0000) 2024-02-19T20:28:18.245 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:18.077+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.078320+0000) 2024-02-19T20:28:18.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:18.308+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:18.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:18.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:18.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:18.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:18.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:18.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:18.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:18.250+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:18.768 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:18.592+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.593137+0000) 2024-02-19T20:28:18.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:18.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:18.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:18.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:18.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:18.751+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:18.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:18.752+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:19.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:19.012+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.013097+0000) 2024-02-19T20:28:19.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:18.775+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:18.776417+0000) 2024-02-19T20:28:19.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:19.301+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:19.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:19.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:19.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:19.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:19.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:19.236+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:19.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:19.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:19.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:19.595+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.596810+0000) 2024-02-19T20:28:19.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:19.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:19.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:19.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:19.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:19.740+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:19.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:19.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:20.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:19.777+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:19.778917+0000) 2024-02-19T20:28:20.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:20.014+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.015709+0000) 2024-02-19T20:28:20.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:20.309+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:20.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:20.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:20.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:20.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:20.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:20.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:20.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:20.256+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:20.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:20.597+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.598804+0000) 2024-02-19T20:28:21.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:21.018+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.019240+0000) 2024-02-19T20:28:21.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:20.780+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:20.781436+0000) 2024-02-19T20:28:21.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:20.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:21.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:20.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:21.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:20.782+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:21.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:20.783+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:21.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:21.287+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:21.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:21.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:21.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:21.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:21.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:21.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:21.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:21.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:21.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:21.599+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.600865+0000) 2024-02-19T20:28:21.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:21.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:21.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:21.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:21.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:21.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:21.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:21.751+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:21.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:21.782+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:21.783626+0000) 2024-02-19T20:28:22.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:22.020+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.021235+0000) 2024-02-19T20:28:22.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:22.278+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:22.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:22.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:22.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:22.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:22.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:22.221+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:22.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:22.222+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:22.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:22.603+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.604064+0000) 2024-02-19T20:28:22.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:22.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:22.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:22.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:22.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:22.750+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:22.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:22.751+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:23.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:23.022+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.023061+0000) 2024-02-19T20:28:23.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:22.784+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:22.785819+0000) 2024-02-19T20:28:23.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:23.089+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64113 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:23.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:23.084+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.085821+0000) 2024-02-19T20:28:23.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:23.085+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.086298+0000) 2024-02-19T20:28:23.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:23.282+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:23.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:23.085+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.086490+0000) 2024-02-19T20:28:23.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:23.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:23.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:23.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:23.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:23.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:23.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:23.250+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:23.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:23.603+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.604238+0000) 2024-02-19T20:28:23.808 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:23.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:23.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:23.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:23.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:23.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:23.809 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:23.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:24.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:24.022+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.023252+0000) 2024-02-19T20:28:24.035 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:23.785+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:23.786057+0000) 2024-02-19T20:28:24.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:24.274+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:24.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:24.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:24.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:24.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:24.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:24.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:24.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:24.298+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:24.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:24.606+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.607757+0000) 2024-02-19T20:28:24.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:24.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:24.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:24.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:24.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:24.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:24.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:24.705+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:25.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:25.024+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.025270+0000) 2024-02-19T20:28:25.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:24.787+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:24.788388+0000) 2024-02-19T20:28:25.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:25.304+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:25.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:25.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:25.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:25.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:25.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:25.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:17:53.728153+0000 (oldest deadline 2024-02-19T20:18:13.728153+0000) 2024-02-19T20:28:25.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:25.312+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:25.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:25.608+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.609840+0000) 2024-02-19T20:28:25.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:25.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:25.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:25.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:25.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:25.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:25.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:25.727+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:26.034 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:25.789+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:25.790732+0000) 2024-02-19T20:28:26.034 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:26.026+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.027691+0000) 2024-02-19T20:28:26.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:26.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:26.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:26.281+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:26.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:26.612+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.613561+0000) 2024-02-19T20:28:26.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:26.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:26.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:26.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:26.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:26.689+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:26.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:26.690+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:27.085 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:27.028+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.029795+0000) 2024-02-19T20:28:27.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:26.791+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:26.792876+0000) 2024-02-19T20:28:27.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:27.285+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:27.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:27.331+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:27.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:27.614+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.615504+0000) 2024-02-19T20:28:27.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:27.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:27.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:27.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:27.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:27.653+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:27.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:27.654+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:28.083 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:28.032+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.033034+0000) 2024-02-19T20:28:28.084 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:27.794+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:27.795887+0000) 2024-02-19T20:28:28.515 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:28.097+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64152 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:28.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:28.094+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.095980+0000) 2024-02-19T20:28:28.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:28.093+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.094971+0000) 2024-02-19T20:28:28.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:28.318+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:28.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:28.093+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.094127+0000) 2024-02-19T20:28:28.517 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:28.242+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:28.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:28.614+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.615718+0000) 2024-02-19T20:28:28.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:28.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:28.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:28.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:28.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:28.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:28.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:28.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:29.085 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:29.032+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.033212+0000) 2024-02-19T20:28:29.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:28.795+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:28.796065+0000) 2024-02-19T20:28:29.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:29.330+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:29.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:29.270+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:29.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:29.616+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.617704+0000) 2024-02-19T20:28:29.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:29.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:29.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:29.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:29.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:29.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:29.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:29.669+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:30.086 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:30.034+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.035647+0000) 2024-02-19T20:28:30.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:29.797+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:29.798036+0000) 2024-02-19T20:28:30.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:30.288+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:30.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:30.228+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:30.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:30.618+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.619789+0000) 2024-02-19T20:28:30.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:30.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:30.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:30.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:30.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:30.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:30.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:30.623+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:31.086 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:31.037+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.039009+0000) 2024-02-19T20:28:31.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:30.799+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:30.800120+0000) 2024-02-19T20:28:31.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:31.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:31.433 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:31.313+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:31.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:31.620+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.621875+0000) 2024-02-19T20:28:31.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:31.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:31.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:31.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:31.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:31.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:31.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:31.644+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:32.086 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:32.041+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.042424+0000) 2024-02-19T20:28:32.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:31.801+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:31.802173+0000) 2024-02-19T20:28:32.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:32.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:32.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:32.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:32.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:32.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:32.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:32.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:32.611 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:32.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:32.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:32.622+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.623694+0000) 2024-02-19T20:28:32.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:32.618+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:32.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:32.804+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:32.805190+0000) 2024-02-19T20:28:33.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:33.100+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.101701+0000) 2024-02-19T20:28:33.285 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:33.104+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64193 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:33.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:33.043+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.044350+0000) 2024-02-19T20:28:33.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:33.101+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.102528+0000) 2024-02-19T20:28:33.286 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:33.102+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.103028+0000) 2024-02-19T20:28:33.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:33.380+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:33.584 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:33.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:33.981 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:33.622+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.623932+0000) 2024-02-19T20:28:33.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:33.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:33.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:33.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:33.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:33.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:18:03.107011+0000 (oldest deadline 2024-02-19T20:18:23.107011+0000) 2024-02-19T20:28:33.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:33.591+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:33.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:33.804+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:33.805406+0000) 2024-02-19T20:28:34.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:34.043+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.044538+0000) 2024-02-19T20:28:34.618 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:34.334+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:34.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:34.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:34.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:34.625+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.626080+0000) 2024-02-19T20:28:34.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:34.638+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:34.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:34.806+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:34.807502+0000) 2024-02-19T20:28:35.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:35.047+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.048054+0000) 2024-02-19T20:28:35.566 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:35.369+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:35.566 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:35.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:35.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:35.627+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.628350+0000) 2024-02-19T20:28:35.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:35.633+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:35.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:35.810+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:35.811125+0000) 2024-02-19T20:28:36.266 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:36.049+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.050034+0000) 2024-02-19T20:28:36.533 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:36.377+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:36.534 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:36.273+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:36.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:36.629+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.630426+0000) 2024-02-19T20:28:36.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:36.683+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:36.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:36.813+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:36.814758+0000) 2024-02-19T20:28:37.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:37.051+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.052067+0000) 2024-02-19T20:28:37.566 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:37.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:37.566 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:37.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:37.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:37.631+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.632256+0000) 2024-02-19T20:28:37.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:37.634+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:37.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:37.816+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:37.817944+0000) 2024-02-19T20:28:38.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:38.108+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.109933+0000) 2024-02-19T20:28:38.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:38.111+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64231 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:38.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:38.108+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.109873+0000) 2024-02-19T20:28:38.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:38.052+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.053909+0000) 2024-02-19T20:28:38.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:38.108+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.109582+0000) 2024-02-19T20:28:38.566 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:38.441+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:38.567 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:38.335+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:38.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:38.631+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.632467+0000) 2024-02-19T20:28:38.982 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:38.676+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:38.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:38.817+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:38.818134+0000) 2024-02-19T20:28:39.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:39.053+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.054137+0000) 2024-02-19T20:28:39.626 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:39.348+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:39.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:39.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:39.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:39.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:39.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:39.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:39.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:39.454+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:39.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:39.633+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.634575+0000) 2024-02-19T20:28:39.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:39.702+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:39.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:39.819+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:39.820175+0000) 2024-02-19T20:28:40.246 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:40.055+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.056131+0000) 2024-02-19T20:28:40.516 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:40.341+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:40.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:40.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:40.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:40.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:40.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:40.405+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:40.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:40.406+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:40.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:40.635+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.636868+0000) 2024-02-19T20:28:40.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:40.725+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:41.050 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:40.821+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:40.822237+0000) 2024-02-19T20:28:41.341 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:41.057+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.058149+0000) 2024-02-19T20:28:41.630 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:41.348+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:41.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:41.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:41.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:41.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:41.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:41.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:41.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:41.428+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:41.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:41.638+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.639052+0000) 2024-02-19T20:28:41.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:41.691+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:41.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:41.823+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:41.824257+0000) 2024-02-19T20:28:42.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:42.060+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.061586+0000) 2024-02-19T20:28:42.632 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:42.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:42.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:42.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:42.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:42.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:42.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:42.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:42.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:42.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:42.982 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:42.640+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.641018+0000) 2024-02-19T20:28:42.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:42.698+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:42.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:42.825+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:42.826100+0000) 2024-02-19T20:28:43.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:43.116+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.117440+0000) 2024-02-19T20:28:43.284 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:43.119+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64272 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:43.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:43.115+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.116919+0000) 2024-02-19T20:28:43.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:43.063+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.064737+0000) 2024-02-19T20:28:43.285 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:43.115+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.116919+0000) 2024-02-19T20:28:43.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:43.369+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:43.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:43.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:43.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:43.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:43.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:43.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:43.634 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:43.416+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:43.983 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:43.640+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.641249+0000) 2024-02-19T20:28:43.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:43.712+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:43.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:43.825+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:43.826289+0000) 2024-02-19T20:28:44.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:44.063+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.064903+0000) 2024-02-19T20:28:44.635 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:44.363+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:44.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:44.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:44.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:44.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:44.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:44.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:44.636 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:44.386+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:44.983 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:44.642+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.643701+0000) 2024-02-19T20:28:44.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:44.678+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:44.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:44.827+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:44.828387+0000) 2024-02-19T20:28:45.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:45.066+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.067018+0000) 2024-02-19T20:28:45.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:45.351+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:45.638 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:45.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:45.638 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:45.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:45.638 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:45.414+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:45.638 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:45.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:45.983 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:45.645+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.646005+0000) 2024-02-19T20:28:45.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:45.660+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:45.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:45.829+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:45.830508+0000) 2024-02-19T20:28:46.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:46.068+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.069146+0000) 2024-02-19T20:28:46.567 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:46.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:46.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:46.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:46.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:46.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:46.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:46.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:46.569 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:46.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:46.569 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:46.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:46.569 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:46.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:46.569 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:46.309+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:46.824 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:46.646+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.647923+0000) 2024-02-19T20:28:46.825 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:46.675+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:47.087 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:47.071+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.071247+0000) 2024-02-19T20:28:47.088 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:46.832+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:46.832613+0000) 2024-02-19T20:28:47.460 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:47.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:47.460 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:47.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:47.460 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:47.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:47.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:47.340+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:47.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:47.666+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:47.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:47.648+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.649801+0000) 2024-02-19T20:28:47.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:47.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:47.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:47.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:47.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:47.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:47.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:47.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:48.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:47.833+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:47.834745+0000) 2024-02-19T20:28:48.473 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:48.128+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64310 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:48.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:48.073+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.073277+0000) 2024-02-19T20:28:48.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:48.125+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.125070+0000) 2024-02-19T20:28:48.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:48.125+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.125397+0000) 2024-02-19T20:28:48.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:48.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:48.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:48.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:48.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:48.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:48.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:48.299+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:48.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:48.125+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.125939+0000) 2024-02-19T20:28:48.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:48.650+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.650011+0000) 2024-02-19T20:28:48.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:48.674+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:48.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:48.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:48.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:48.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:48.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:48.481+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:48.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:48.482+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:49.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:48.834+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:48.834939+0000) 2024-02-19T20:28:49.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:49.072+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.073478+0000) 2024-02-19T20:28:49.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:49.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:49.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:49.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:49.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:49.269+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:49.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:49.269+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:49.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:49.651+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.652126+0000) 2024-02-19T20:28:49.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:49.653+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:49.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:49.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:49.787 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:49.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:49.788 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:49.530+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:49.788 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:49.531+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:50.067 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:49.836+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:49.837011+0000) 2024-02-19T20:28:50.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:50.075+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.076058+0000) 2024-02-19T20:28:50.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:50.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:50.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:50.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:50.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:50.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:50.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:50.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:50.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:50.696+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:50.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:50.653+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.654160+0000) 2024-02-19T20:28:50.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:50.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:50.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:50.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:50.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:50.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:50.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:50.540+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:51.070 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:50.838+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:50.839128+0000) 2024-02-19T20:28:51.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:51.077+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.078127+0000) 2024-02-19T20:28:51.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:51.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:51.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:51.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:51.511 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:51.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:51.512 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:51.311+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:51.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:51.701+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:51.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:51.656+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.657911+0000) 2024-02-19T20:28:51.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:51.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:51.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:51.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:51.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:51.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:51.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:51.518+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:52.072 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:51.841+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:51.842752+0000) 2024-02-19T20:28:52.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:52.079+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.080186+0000) 2024-02-19T20:28:52.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:52.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:52.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:52.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:52.489 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:52.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:52.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:52.311+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:52.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:52.660+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.661119+0000) 2024-02-19T20:28:52.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:52.705+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:52.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:52.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:52.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:52.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:52.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:52.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:52.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:52.497+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:53.074 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:52.843+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:52.844756+0000) 2024-02-19T20:28:53.470 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:53.134+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64351 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:53.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:53.132+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.133316+0000) 2024-02-19T20:28:53.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:53.081+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.082694+0000) 2024-02-19T20:28:53.471 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:53.131+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.132643+0000) 2024-02-19T20:28:53.471 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:53.131+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.132879+0000) 2024-02-19T20:28:53.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:53.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:53.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:53.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:53.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:53.320+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:53.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:53.321+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:53.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:53.689+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:53.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:53.660+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.661280+0000) 2024-02-19T20:28:53.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:53.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:53.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:53.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:53.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:53.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:53.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:53.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:54.074 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:53.843+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:53.844961+0000) 2024-02-19T20:28:54.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:54.081+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.082891+0000) 2024-02-19T20:28:54.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:54.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:54.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:54.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:54.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:54.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:54.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:54.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:54.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:54.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:54.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:54.318+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:54.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:54.662+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.663569+0000) 2024-02-19T20:28:54.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:54.680+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:54.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:54.496+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:54.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:54.497+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:55.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:54.846+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:54.847066+0000) 2024-02-19T20:28:55.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:55.084+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.085464+0000) 2024-02-19T20:28:55.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:55.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:55.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:55.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:55.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:55.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:55.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:55.320+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:55.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:55.664+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.665643+0000) 2024-02-19T20:28:55.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:55.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:55.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:55.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:55.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:55.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:55.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:55.534+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:55.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:55.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:55.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:55.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:55.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:55.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:55.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:55.696+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:56.079 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:55.848+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:55.849219+0000) 2024-02-19T20:28:56.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:56.086+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.087604+0000) 2024-02-19T20:28:56.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:56.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:56.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:56.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:56.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:56.312+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:56.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:56.312+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:56.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:56.666+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.667790+0000) 2024-02-19T20:28:56.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:56.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:56.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:56.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:56.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:56.561+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:56.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:56.562+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:56.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:56.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:56.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:56.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:56.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:56.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:56.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:56.681+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:57.081 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:56.850+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:56.851406+0000) 2024-02-19T20:28:57.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:57.088+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.089982+0000) 2024-02-19T20:28:57.518 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:57.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:57.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:57.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:57.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:57.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:57.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:57.265+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:57.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:57.668+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.669649+0000) 2024-02-19T20:28:57.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:57.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:57.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:57.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:57.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:57.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:57.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:57.595+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:57.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:57.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:57.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:57.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:57.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:57.707+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:57.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:57.708+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:58.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:57.852+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:57.853336+0000) 2024-02-19T20:28:58.518 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:28:58.142+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64389 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:28:58.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:58.139+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.140469+0000) 2024-02-19T20:28:58.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:58.090+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.091904+0000) 2024-02-19T20:28:58.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:58.139+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.140508+0000) 2024-02-19T20:28:58.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:58.139+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.140372+0000) 2024-02-19T20:28:58.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:58.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:58.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:58.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:58.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:58.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:58.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:58.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:58.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:58.668+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.669878+0000) 2024-02-19T20:28:58.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:58.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:58.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:58.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:58.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:58.585+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:58.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:58.586+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:58.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:58.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:58.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:58.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:58.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:58.746+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:58.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:58.747+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:28:59.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:58.852+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:58.853540+0000) 2024-02-19T20:28:59.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:59.091+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.092051+0000) 2024-02-19T20:28:59.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:59.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:59.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:59.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:59.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:59.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:28:59.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:59.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:59.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:28:59.670+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.671910+0000) 2024-02-19T20:28:59.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:59.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:59.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:59.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:59.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:59.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:28:59.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:28:59.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:28:59.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:59.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:59.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:59.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:59.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:59.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:28:59.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:59.744+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:00.086 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:28:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:28:59.854+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:28:59.855940+0000) 2024-02-19T20:29:00.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:00.093+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.094506+0000) 2024-02-19T20:29:00.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:00.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:00.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:00.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:00.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:00.295+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:00.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:00.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:00.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:00.672+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.673937+0000) 2024-02-19T20:29:00.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:00.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:00.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:00.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:00.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:00.711+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:00.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:00.712+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:00.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:00.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:00.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:00.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:00.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:00.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:00.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:00.616+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:01.088 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:00.857+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:00.858486+0000) 2024-02-19T20:29:01.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:01.095+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.096595+0000) 2024-02-19T20:29:01.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:01.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:01.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:01.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:01.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:01.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:01.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:01.266+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:01.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:01.676+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.677635+0000) 2024-02-19T20:29:01.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:01.573+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:01.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:01.573+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:01.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:01.573+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:01.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:01.573+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:01.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:01.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:01.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:01.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:01.710+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:01.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:01.712+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:02.089 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:01.859+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:01.860488+0000) 2024-02-19T20:29:02.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:02.097+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.098658+0000) 2024-02-19T20:29:02.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:02.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:02.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:02.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:02.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:02.274+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:02.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:02.275+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:02.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:02.678+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.679519+0000) 2024-02-19T20:29:02.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:02.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:02.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:02.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:02.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:02.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:02.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:02.676+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:02.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:02.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:02.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:02.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:02.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:02.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:02.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:02.572+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:03.089 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:02.861+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:02.862366+0000) 2024-02-19T20:29:03.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:03.150+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64430 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:03.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:03.146+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.147703+0000) 2024-02-19T20:29:03.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:03.099+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.100474+0000) 2024-02-19T20:29:03.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:03.146+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.147955+0000) 2024-02-19T20:29:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:03.146+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.147306+0000) 2024-02-19T20:29:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:03.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:03.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:03.263+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:03.264+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:03.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:03.678+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.679723+0000) 2024-02-19T20:29:03.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:03.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:03.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:03.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:03.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:03.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:03.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:03.590+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:03.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:03.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:03.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:03.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:03.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:03.698+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:03.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:03.699+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:04.089 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:03.861+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:03.862563+0000) 2024-02-19T20:29:04.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:04.099+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.100661+0000) 2024-02-19T20:29:04.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:04.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:04.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:04.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:04.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:04.290+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:04.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:04.290+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:04.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:04.680+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.681750+0000) 2024-02-19T20:29:04.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:04.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:04.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:04.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:04.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:04.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:04.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:04.620+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:04.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:04.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:04.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:04.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:04.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:04.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:04.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:04.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:05.089 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:04.863+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:04.864719+0000) 2024-02-19T20:29:05.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:05.103+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.104251+0000) 2024-02-19T20:29:05.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:05.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:05.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:05.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:05.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:05.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:05.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:05.245+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:05.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:05.682+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.683958+0000) 2024-02-19T20:29:05.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:05.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:05.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:05.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:05.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:05.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:05.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:05.659+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:05.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:05.573+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:05.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:05.573+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:05.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:05.573+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:05.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:05.574+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:06.089 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:05.865+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:05.866779+0000) 2024-02-19T20:29:06.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:06.106+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.107922+0000) 2024-02-19T20:29:06.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:06.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:06.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:06.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:06.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:06.284+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:06.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:06.285+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:06.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:06.685+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.686058+0000) 2024-02-19T20:29:06.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:06.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:06.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:06.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:06.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:06.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:06.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:06.681+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:06.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:06.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:06.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:06.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:06.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:06.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:06.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:06.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:07.089 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:06.869+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:06.870407+0000) 2024-02-19T20:29:07.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:07.109+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.110065+0000) 2024-02-19T20:29:07.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:07.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:07.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:07.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:07.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:07.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:07.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:07.302+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:07.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:07.686+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.687971+0000) 2024-02-19T20:29:07.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:07.652+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:07.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:07.652+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:07.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:07.652+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:07.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:07.653+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:07.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:07.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:07.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:07.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:07.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:07.576+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:07.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:07.577+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:08.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:07.871+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:07.872360+0000) 2024-02-19T20:29:08.519 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:08.157+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64468 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:08.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:08.154+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.155849+0000) 2024-02-19T20:29:08.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:08.110+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.111897+0000) 2024-02-19T20:29:08.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:08.155+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.156128+0000) 2024-02-19T20:29:08.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:08.154+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.155158+0000) 2024-02-19T20:29:08.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:08.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:08.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:08.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:08.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:08.253+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:08.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:08.253+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:08.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:08.687+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.688157+0000) 2024-02-19T20:29:08.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:08.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:08.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:08.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:08.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:08.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:08.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:08.612+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:08.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:08.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:08.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:08.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:08.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:08.528+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:08.786 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:08.529+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:09.089 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:08.871+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:08.872599+0000) 2024-02-19T20:29:09.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:09.111+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.112061+0000) 2024-02-19T20:29:09.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:09.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:09.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:09.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:09.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:09.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:09.491 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:09.230+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:09.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:09.690+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.691551+0000) 2024-02-19T20:29:09.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:09.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:09.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:09.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:09.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:09.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:09.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:09.500+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:09.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:09.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:09.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:09.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:09.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:09.585+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:09.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:09.586+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:10.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:09.875+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:09.876095+0000) 2024-02-19T20:29:10.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:10.113+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.114286+0000) 2024-02-19T20:29:10.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:10.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:10.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:10.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:10.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:10.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:10.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:10.273+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:10.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:10.693+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.694979+0000) 2024-02-19T20:29:10.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:10.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:10.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:10.500+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:10.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:10.501+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:10.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:10.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:10.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:10.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:10.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:10.549+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:10.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:10.550+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:11.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:10.878+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:10.879566+0000) 2024-02-19T20:29:11.369 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:11.115+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.116434+0000) 2024-02-19T20:29:11.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:11.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:11.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:11.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:11.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:11.273+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:11.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:11.274+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:11.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:11.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:11.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:11.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:11.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:11.580+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:11.691 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:11.581+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:11.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:11.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:11.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:11.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:11.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:11.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:11.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:11.475+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:11.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:11.697+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.698657+0000) 2024-02-19T20:29:11.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:11.882+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:11.883051+0000) 2024-02-19T20:29:12.236 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:12.117+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.118504+0000) 2024-02-19T20:29:12.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:12.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:12.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:12.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:12.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:12.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:12.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:12.433+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:12.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:12.244+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:12.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:12.244+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:12.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:12.244+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:12.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:12.244+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:12.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:12.701+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.702079+0000) 2024-02-19T20:29:12.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:12.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:12.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:12.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:12.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:12.604+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:12.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:12.605+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:13.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:12.885+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:12.886275+0000) 2024-02-19T20:29:13.409 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:13.165+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64509 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:13.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:13.119+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.120410+0000) 2024-02-19T20:29:13.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:13.161+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.162440+0000) 2024-02-19T20:29:13.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:13.163+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.164422+0000) 2024-02-19T20:29:13.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:13.161+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.162880+0000) 2024-02-19T20:29:13.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:13.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:13.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:13.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:13.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:13.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:13.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:13.216+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:13.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:13.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:13.694 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:13.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:13.694 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:13.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:13.694 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:13.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:13.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:13.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:13.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:13.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:13.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:13.601+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:13.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:13.601+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:13.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:13.701+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.702252+0000) 2024-02-19T20:29:13.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:13.885+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:13.886432+0000) 2024-02-19T20:29:14.253 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:14.119+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.120616+0000) 2024-02-19T20:29:14.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:14.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:14.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:14.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:14.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:14.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:14.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:14.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:14.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:14.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:14.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:14.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:14.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:14.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:14.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:14.262+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:14.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:14.703+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.704750+0000) 2024-02-19T20:29:14.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:14.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:14.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:14.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:14.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:14.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:14.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:14.636+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:15.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:14.888+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:14.889862+0000) 2024-02-19T20:29:15.409 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:15.121+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.122733+0000) 2024-02-19T20:29:15.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:15.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:15.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:15.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:15.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:15.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:15.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:15.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:15.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:15.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:15.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:15.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:15.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:15.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:15.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:15.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:15.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:15.622+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:15.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:15.622+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:15.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:15.622+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:15.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:15.623+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:15.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:15.705+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.706883+0000) 2024-02-19T20:29:15.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:15.892+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:15.893157+0000) 2024-02-19T20:29:16.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:16.124+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.125242+0000) 2024-02-19T20:29:16.570 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:16.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:16.570 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:16.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:16.570 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:16.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:16.570 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:16.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:16.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:16.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:16.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:16.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:16.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:16.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:16.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:16.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:16.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:16.577+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:16.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:16.577+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:16.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:16.577+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:16.887 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:16.708+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.709453+0000) 2024-02-19T20:29:16.887 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:16.578+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:17.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:17.126+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.127646+0000) 2024-02-19T20:29:17.171 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:16.894+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:16.895592+0000) 2024-02-19T20:29:17.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:17.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:17.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:17.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:17.433 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:17.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:17.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:17.303+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:17.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:17.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:17.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:17.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:17.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:17.440+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:17.703 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:17.441+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:17.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:17.586+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:17.703 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:17.586+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:17.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:17.586+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:17.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:17.587+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:17.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:17.710+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.711377+0000) 2024-02-19T20:29:17.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:17.897+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:17.898749+0000) 2024-02-19T20:29:18.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:18.169+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.170099+0000) 2024-02-19T20:29:18.283 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:18.174+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64547 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:18.284 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:18.170+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.171090+0000) 2024-02-19T20:29:18.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:18.129+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.130135+0000) 2024-02-19T20:29:18.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:18.169+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.170067+0000) 2024-02-19T20:29:18.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:18.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:18.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:18.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:18.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:18.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:18.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:18.297+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:18.571 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:18.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:18.571 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:18.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:18.571 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:18.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:18.571 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:18.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:18.890 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:18.710+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.711560+0000) 2024-02-19T20:29:18.891 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:18.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:18.891 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:18.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:18.891 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:18.623+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:18.891 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:18.624+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:19.173 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:19.129+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.130318+0000) 2024-02-19T20:29:19.174 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:18.897+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:18.898963+0000) 2024-02-19T20:29:19.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:19.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:19.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:19.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:19.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:19.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:19.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:19.297+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:19.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:19.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:19.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:19.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:19.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:19.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:19.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:19.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:19.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:19.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:19.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:19.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:19.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:19.654+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:19.707 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:19.654+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:19.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:19.900+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.901271+0000) 2024-02-19T20:29:19.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:19.713+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:19.714159+0000) 2024-02-19T20:29:20.249 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:20.131+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.132320+0000) 2024-02-19T20:29:20.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:20.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:20.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:20.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:20.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:20.462+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:20.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:20.463+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:20.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:20.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:20.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:20.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:20.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:20.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:20.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:20.282+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:20.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:20.715+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.716706+0000) 2024-02-19T20:29:20.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:20.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:20.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:20.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:20.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:20.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:20.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:20.668+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:21.090 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:20.903+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:20.904939+0000) 2024-02-19T20:29:21.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:21.133+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.134386+0000) 2024-02-19T20:29:21.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:21.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:21.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:21.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:21.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:21.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:21.459 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:21.324+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:21.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:21.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:21.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:21.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:21.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:21.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:21.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:21.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:21.710 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:21.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:21.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:21.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:21.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:21.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:21.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:21.696+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:21.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:21.717+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.718760+0000) 2024-02-19T20:29:21.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:21.907+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:21.908418+0000) 2024-02-19T20:29:22.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:22.135+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.136449+0000) 2024-02-19T20:29:22.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:22.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:22.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:22.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:22.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:22.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:22.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:22.372+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:22.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:22.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:22.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:22.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:22.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:22.489+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:22.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:22.490+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:22.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:22.720+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.721988+0000) 2024-02-19T20:29:22.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:22.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:22.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:22.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:22.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:22.720+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:22.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:22.721+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:22.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:22.910+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:22.911469+0000) 2024-02-19T20:29:23.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:23.178+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.179054+0000) 2024-02-19T20:29:23.283 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:23.181+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64588 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:23.284 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:23.179+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.180459+0000) 2024-02-19T20:29:23.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:23.137+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.138365+0000) 2024-02-19T20:29:23.284 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:23.177+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.178813+0000) 2024-02-19T20:29:23.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:23.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:23.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:23.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:23.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:23.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:23.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:23.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:23.571 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:23.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:23.571 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:23.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:23.572 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:23.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:23.572 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:23.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:23.903 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:23.721+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.722202+0000) 2024-02-19T20:29:23.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:23.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:23.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:23.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:23.903 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:23.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:23.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:23.760+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:23.904 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:23.910+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:23.911704+0000) 2024-02-19T20:29:24.178 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:24.137+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.138581+0000) 2024-02-19T20:29:24.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:24.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:24.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:24.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:24.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:24.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:24.462 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:24.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:24.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:24.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:24.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:24.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:24.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:24.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:24.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:24.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:24.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:24.723+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.724825+0000) 2024-02-19T20:29:24.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:24.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:24.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:24.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:24.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:24.733+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:24.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:24.734+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:24.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:24.912+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:24.913870+0000) 2024-02-19T20:29:25.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:25.139+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.140577+0000) 2024-02-19T20:29:25.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:25.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:25.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:25.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:25.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:25.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:25.678 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:25.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:25.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:25.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:25.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:25.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:25.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:25.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:25.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:25.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:25.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:25.725+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.726947+0000) 2024-02-19T20:29:25.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:25.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:25.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:25.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:25.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:25.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:25.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:25.686+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:25.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:25.916+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:25.917395+0000) 2024-02-19T20:29:26.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:26.141+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.142649+0000) 2024-02-19T20:29:26.636 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:26.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:26.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:26.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:26.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:26.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:26.637 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:26.351+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:26.637 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:26.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:26.637 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:26.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:26.638 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:26.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:26.638 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:26.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:26.910 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:26.727+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.728950+0000) 2024-02-19T20:29:26.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:26.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:26.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:26.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:26.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:26.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:26.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:26.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:27.180 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:27.143+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.144875+0000) 2024-02-19T20:29:27.180 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:26.918+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:26.919463+0000) 2024-02-19T20:29:27.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:27.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:27.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:27.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:27.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:27.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:27.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:27.359+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:27.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:27.730+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.731273+0000) 2024-02-19T20:29:27.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:27.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:27.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:27.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:27.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:27.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:27.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:27.691+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:27.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:27.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:27.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:27.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:27.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:27.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:27.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:27.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:28.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:27.920+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:27.921388+0000) 2024-02-19T20:29:28.342 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:28.189+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64626 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:28.342 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:28.185+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.186479+0000) 2024-02-19T20:29:28.343 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:28.145+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.146815+0000) 2024-02-19T20:29:28.343 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:28.186+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.187240+0000) 2024-02-19T20:29:28.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:28.186+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.187191+0000) 2024-02-19T20:29:28.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:28.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:28.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:28.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:28.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:28.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:28.344 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:28.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:28.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:28.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:28.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:28.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:28.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:28.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:28.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:28.567+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:28.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:28.730+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.731471+0000) 2024-02-19T20:29:28.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:28.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:28.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:28.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:28.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:28.690+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:28.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:28.691+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:28.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:28.920+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:28.921563+0000) 2024-02-19T20:29:29.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:29.145+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.147011+0000) 2024-02-19T20:29:29.571 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:29.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:29.571 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:29.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:29.572 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:29.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:29.572 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:29.572+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:29.572 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:29.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:29.572 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:29.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:29.572 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:29.326+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:29.572 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:29.327+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:29.916 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:29.733+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.734093+0000) 2024-02-19T20:29:29.916 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:29.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:29.916 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:29.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:29.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:29.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:29.917 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:29.693+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:30.185 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:30.149+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.150509+0000) 2024-02-19T20:29:30.185 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:29.923+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:29.924999+0000) 2024-02-19T20:29:30.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:30.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:30.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:30.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:30.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:30.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:30.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:30.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:30.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:30.735+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.736522+0000) 2024-02-19T20:29:30.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:30.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:30.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:30.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:30.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:30.727+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:30.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:30.728+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:30.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:30.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:30.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:30.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:30.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:30.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:30.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:30.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:31.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:30.927+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:30.928537+0000) 2024-02-19T20:29:31.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:31.153+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.154062+0000) 2024-02-19T20:29:31.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:31.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:31.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:31.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:31.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:31.278+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:31.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:31.279+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:31.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:31.737+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.738575+0000) 2024-02-19T20:29:31.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:31.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:31.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:31.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:31.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:31.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:31.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:31.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:31.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:31.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:31.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:31.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:31.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:31.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:31.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:31.739+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:32.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:31.930+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:31.931819+0000) 2024-02-19T20:29:32.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:32.156+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.157588+0000) 2024-02-19T20:29:32.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:32.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:32.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:32.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:32.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:32.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:32.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:32.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:32.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:32.740+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.741166+0000) 2024-02-19T20:29:32.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:32.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:32.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:32.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:32.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:32.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:32.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:32.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:32.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:32.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:32.787 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:32.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:32.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:32.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:32.788 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:32.718+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:33.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:32.933+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:32.934275+0000) 2024-02-19T20:29:33.522 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:33.197+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64667 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:33.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:33.193+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.194336+0000) 2024-02-19T20:29:33.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:33.158+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.159731+0000) 2024-02-19T20:29:33.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:33.193+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.194668+0000) 2024-02-19T20:29:33.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:33.193+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.194875+0000) 2024-02-19T20:29:33.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:33.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:33.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:33.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:33.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:33.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:33.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:33.266+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:33.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:33.741+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.741353+0000) 2024-02-19T20:29:33.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:33.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:33.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:33.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:33.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:33.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:33.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:33.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:33.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:33.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:33.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:33.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:33.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:33.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:33.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:33.765+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:34.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:33.934+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:33.934442+0000) 2024-02-19T20:29:34.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:34.158+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.160002+0000) 2024-02-19T20:29:34.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:34.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:34.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:34.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:34.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:34.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:34.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:34.298+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:34.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:34.743+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.743404+0000) 2024-02-19T20:29:34.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:34.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:34.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:34.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:34.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:34.619+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:34.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:34.620+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:34.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:34.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:34.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:34.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:34.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:34.765+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:34.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:34.766+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:35.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:34.936+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:34.936473+0000) 2024-02-19T20:29:35.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:35.162+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.162034+0000) 2024-02-19T20:29:35.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:35.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:35.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:35.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:35.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:35.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:35.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:35.279+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:35.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:35.744+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.745421+0000) 2024-02-19T20:29:35.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:35.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:35.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:35.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:35.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:35.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:35.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:35.738+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:35.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:35.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:35.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:35.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:35.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:35.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:35.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:35.609+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:36.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:35.938+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:35.940003+0000) 2024-02-19T20:29:36.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:36.163+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.164398+0000) 2024-02-19T20:29:36.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:36.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:36.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:36.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:36.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:36.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:36.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:36.257+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:36.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:36.746+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.747552+0000) 2024-02-19T20:29:36.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:36.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:36.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:36.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:36.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:36.562+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:36.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:36.563+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:36.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:36.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:36.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:36.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:36.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:36.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:36.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:36.692+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:37.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:36.942+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:36.943536+0000) 2024-02-19T20:29:37.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:37.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:37.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:37.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:37.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:37.285+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:37.508 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:37.286+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:37.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:37.167+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.168281+0000) 2024-02-19T20:29:37.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:37.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:37.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:37.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:37.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:37.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:37.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:37.748+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.749498+0000) 2024-02-19T20:29:37.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:37.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:37.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:37.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:37.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:37.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:37.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:37.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:37.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:37.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:38.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:37.945+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:37.946065+0000) 2024-02-19T20:29:38.477 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:38.205+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64705 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:38.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:38.201+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.202232+0000) 2024-02-19T20:29:38.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:38.169+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.170682+0000) 2024-02-19T20:29:38.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:38.201+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.202356+0000) 2024-02-19T20:29:38.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:38.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:38.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:38.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:38.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:38.485+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:38.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:38.202+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.203032+0000) 2024-02-19T20:29:38.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:38.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:38.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:38.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:38.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:38.329+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:38.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:38.330+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:38.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:38.486+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:38.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:38.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:38.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:38.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:38.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:38.621+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:38.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:38.621+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:39.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:38.945+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.946251+0000) 2024-02-19T20:29:39.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:38.748+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:38.749671+0000) 2024-02-19T20:29:39.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:39.169+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.170928+0000) 2024-02-19T20:29:39.471 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:39.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:39.471 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:39.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:39.471 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:39.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:39.471 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:39.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:39.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:39.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:39.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:39.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:39.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:39.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:39.491 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:39.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:39.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:39.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:39.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:39.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:39.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:39.607+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:39.743 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:39.607+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:40.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:39.947+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.948350+0000) 2024-02-19T20:29:40.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:39.751+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:39.752174+0000) 2024-02-19T20:29:40.456 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:40.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:40.457 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:40.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:40.457 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:40.327+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:40.457 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:40.327+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:40.457 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:40.171+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.172956+0000) 2024-02-19T20:29:40.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:40.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:40.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:40.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:40.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:40.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:40.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:40.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:40.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:40.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:40.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:40.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:40.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:40.567+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:40.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:40.567+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:41.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:40.753+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.754380+0000) 2024-02-19T20:29:41.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:40.949+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:40.950889+0000) 2024-02-19T20:29:41.346 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:41.174+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.175637+0000) 2024-02-19T20:29:41.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:41.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:41.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:41.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:41.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:41.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:41.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:41.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:41.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:41.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:41.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:41.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:41.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:41.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:41.749 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:41.572+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:41.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:41.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:41.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:41.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:41.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:41.355+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:41.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:41.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:42.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:41.755+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.756624+0000) 2024-02-19T20:29:42.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:41.952+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:41.953063+0000) 2024-02-19T20:29:42.460 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:42.177+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.178221+0000) 2024-02-19T20:29:42.460 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:42.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:42.461 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:42.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:42.461 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:42.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:42.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:42.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:42.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:42.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:42.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:42.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:42.461 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:42.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:42.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:42.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:42.616+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:42.616+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:42.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:42.616+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:42.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:42.617+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:43.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:42.954+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.955149+0000) 2024-02-19T20:29:43.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:42.758+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:42.759254+0000) 2024-02-19T20:29:43.438 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:43.212+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64746 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:43.438 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:43.209+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.210778+0000) 2024-02-19T20:29:43.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:43.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:43.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:43.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:43.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:43.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:43.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:43.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:43.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:43.179+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.180586+0000) 2024-02-19T20:29:43.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:43.208+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.209724+0000) 2024-02-19T20:29:43.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:43.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:43.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:43.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:43.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:43.446+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:43.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:43.209+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.210119+0000) 2024-02-19T20:29:43.750 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:43.447+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:43.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:43.594+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:43.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:43.594+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:43.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:43.594+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:43.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:43.595+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:44.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:43.758+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.759514+0000) 2024-02-19T20:29:44.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:43.954+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:43.955410+0000) 2024-02-19T20:29:44.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:44.179+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.180770+0000) 2024-02-19T20:29:44.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:44.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:44.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:44.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:44.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:44.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:44.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:44.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:44.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:44.318+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:44.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:44.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:44.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:44.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:44.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:44.414+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:44.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:44.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:44.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:44.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:44.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:44.611+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:44.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:44.612+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:45.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:44.956+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.957423+0000) 2024-02-19T20:29:45.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:44.761+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:44.762068+0000) 2024-02-19T20:29:45.358 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:45.183+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.184233+0000) 2024-02-19T20:29:45.359 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:45.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:45.359 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:45.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:45.359 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:45.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:45.359 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:45.348+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:45.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:45.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:45.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:45.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:45.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:45.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:45.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:45.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:45.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:45.763+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.764538+0000) 2024-02-19T20:29:45.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:45.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:45.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:45.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:45.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:45.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:45.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:45.634+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:46.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:46.186+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.187675+0000) 2024-02-19T20:29:46.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:45.958+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:45.959613+0000) 2024-02-19T20:29:46.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:46.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:46.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:46.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:46.628 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:46.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:46.629 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:46.405+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:46.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:46.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:46.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:46.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:46.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:46.382+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:46.629 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:46.382+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:46.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:46.765+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.766745+0000) 2024-02-19T20:29:46.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:46.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:46.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:46.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:46.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:46.636+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:46.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:46.636+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:47.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:46.960+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:46.961877+0000) 2024-02-19T20:29:47.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:47.190+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.191056+0000) 2024-02-19T20:29:47.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:47.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:47.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:47.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:47.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:47.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:47.627 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:47.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:47.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:47.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:47.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:47.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:47.628 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:47.402+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:47.628 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:47.403+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:47.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:47.769+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.770146+0000) 2024-02-19T20:29:47.955 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:47.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:47.955 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:47.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:47.956 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:47.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:47.956 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:47.635+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:48.205 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:48.193+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.194345+0000) 2024-02-19T20:29:48.206 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:47.963+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:47.964017+0000) 2024-02-19T20:29:48.523 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:48.220+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64786 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:48.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:48.216+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.217694+0000) 2024-02-19T20:29:48.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:48.217+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.218421+0000) 2024-02-19T20:29:48.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:48.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:48.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:48.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:48.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:48.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:48.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:48.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:48.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:48.218+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.219063+0000) 2024-02-19T20:29:48.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:48.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:48.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:48.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:48.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:48.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:48.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:48.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:48.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:48.769+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.770339+0000) 2024-02-19T20:29:48.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:48.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:48.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:48.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:48.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:48.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:48.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:48.632+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:49.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:48.963+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:48.964217+0000) 2024-02-19T20:29:49.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:49.193+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.194534+0000) 2024-02-19T20:29:49.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:49.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:49.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:49.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:49.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:49.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:49.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:49.428+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:49.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:49.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:49.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:49.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:49.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:49.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:49.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:49.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:49.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:49.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:49.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:49.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:49.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:49.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:49.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:49.671+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:50.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:49.771+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.772295+0000) 2024-02-19T20:29:50.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:49.965+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:49.966219+0000) 2024-02-19T20:29:50.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:50.196+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.197099+0000) 2024-02-19T20:29:50.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:50.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:50.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:50.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:50.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:50.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:50.686 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:50.445+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:50.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:50.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:50.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:50.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:50.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:50.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:50.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:50.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:50.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:50.773+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.774616+0000) 2024-02-19T20:29:50.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:50.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:50.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:50.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:50.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:50.693+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:50.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:50.694+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:51.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:50.967+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:50.968224+0000) 2024-02-19T20:29:51.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:51.199+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.200521+0000) 2024-02-19T20:29:51.573 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:51.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:51.574 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:51.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:51.574 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:51.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:51.574 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:51.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:51.574 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:51.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:51.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:51.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:51.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:51.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:51.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:51.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:51.952 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:51.776+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.777227+0000) 2024-02-19T20:29:51.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:51.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:51.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:51.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:51.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:51.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:51.953 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:51.686+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:52.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:51.969+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:51.970861+0000) 2024-02-19T20:29:52.283 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:52.202+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.203150+0000) 2024-02-19T20:29:52.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:52.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:52.647 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:52.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:52.648 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:52.385+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:52.648 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:52.386+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:52.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:52.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:52.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:52.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:52.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:52.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:52.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:52.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:52.649 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:52.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:52.649 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:52.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:52.649 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:52.655+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:52.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:52.656+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:52.953 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:52.778+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.779232+0000) 2024-02-19T20:29:53.213 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:53.204+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.205505+0000) 2024-02-19T20:29:53.214 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:52.972+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:52.973330+0000) 2024-02-19T20:29:53.481 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:53.226+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.227211+0000) 2024-02-19T20:29:53.481 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:53.228+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64827 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:53.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:53.225+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.226919+0000) 2024-02-19T20:29:53.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:53.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:53.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:53.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:53.482 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:53.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:53.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:53.224+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.225626+0000) 2024-02-19T20:29:53.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:53.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:53.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:53.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:53.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:53.422+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:53.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:53.424+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:53.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:53.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:53.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:53.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:53.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:53.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:53.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:53.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:53.771 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:53.639+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:54.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:53.972+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.973515+0000) 2024-02-19T20:29:54.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:53.778+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:53.779401+0000) 2024-02-19T20:29:54.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:54.204+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.205713+0000) 2024-02-19T20:29:54.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:54.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:54.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:54.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:54.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:54.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:54.773 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:54.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:54.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:54.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:54.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:54.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:54.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:54.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:54.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:54.627+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:54.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:54.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:54.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:54.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:54.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:54.530+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:54.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:54.530+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:55.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:54.974+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.975556+0000) 2024-02-19T20:29:55.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:54.780+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:54.781811+0000) 2024-02-19T20:29:55.449 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:55.206+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.207788+0000) 2024-02-19T20:29:55.449 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:55.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:55.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:55.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:55.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:55.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:55.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:55.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:55.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:55.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:55.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:55.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:55.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:55.583+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:55.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:55.584+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:55.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:55.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:55.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:55.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:55.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:55.534+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:55.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:55.534+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:56.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:55.783+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.784233+0000) 2024-02-19T20:29:56.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:55.976+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:55.977597+0000) 2024-02-19T20:29:56.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:56.208+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.209834+0000) 2024-02-19T20:29:56.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:56.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:56.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:56.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:56.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:56.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:56.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:56.449+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:56.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:56.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:56.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:56.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:56.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:56.504+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:56.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:56.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:56.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:56.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:56.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:56.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:56.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:56.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:56.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:56.606+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:57.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:56.785+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.786357+0000) 2024-02-19T20:29:57.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:56.978+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:56.979803+0000) 2024-02-19T20:29:57.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:57.211+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.212058+0000) 2024-02-19T20:29:57.457 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:57.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:57.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:57.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:57.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:57.464+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:57.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:57.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:57.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:57.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:57.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:57.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:57.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:57.640+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:57.785 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:57.640+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:57.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:57.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:57.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:57.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:57.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:57.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:57.786 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:57.506+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:58.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:57.981+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.982002+0000) 2024-02-19T20:29:58.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:57.787+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:57.788897+0000) 2024-02-19T20:29:58.345 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:29:58.236+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64866 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:29:58.345 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:58.213+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.214087+0000) 2024-02-19T20:29:58.345 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:58.233+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.234115+0000) 2024-02-19T20:29:58.346 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:58.235+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.236599+0000) 2024-02-19T20:29:58.346 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:58.233+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.234202+0000) 2024-02-19T20:29:58.621 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:58.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:58.621 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:58.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:58.622 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:58.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:58.622 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:58.499+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:58.622 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:58.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:58.622 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:58.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:58.622 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:58.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:58.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:58.532+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:58.623 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:58.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:58.623 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:58.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:58.623 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:58.629+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:58.957 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:58.788+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.789106+0000) 2024-02-19T20:29:58.958 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:58.630+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:59.282 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:59.213+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.214287+0000) 2024-02-19T20:29:59.283 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:58.981+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:58.982227+0000) 2024-02-19T20:29:59.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:59.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:59.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:59.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:59.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:59.546+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:29:59.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:29:59.546+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:29:59.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:59.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:59.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:59.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:59.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:59.625+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:29:59.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:59.626+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:29:59.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:59.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:59.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:59.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:59.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:59.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:29:59.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:59.545+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:00.033 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:29:59.984+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.985104+0000) 2024-02-19T20:30:00.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:29:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:29:59.790+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:29:59.791646+0000) 2024-02-19T20:30:00.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:00.215+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.216319+0000) 2024-02-19T20:30:00.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:00.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:00.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:00.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:00.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:00.541+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:00.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:00.542+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:00.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:00.664+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:00.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:00.664+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:00.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:00.664+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:00.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:00.665+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:00.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:00.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:00.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:00.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:00.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:00.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:00.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:00.579+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:01.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:00.986+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.987084+0000) 2024-02-19T20:30:01.095 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:00.793+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:00.794128+0000) 2024-02-19T20:30:01.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:01.217+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.218555+0000) 2024-02-19T20:30:01.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:01.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:01.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:01.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:01.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:01.591+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:01.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:01.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:01.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:01.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:01.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:01.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:01.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:01.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:01.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:01.663+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:01.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:01.595+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:01.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:01.595+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:01.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:01.595+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:01.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:01.596+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:02.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:01.795+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.796174+0000) 2024-02-19T20:30:02.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:01.988+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:01.989602+0000) 2024-02-19T20:30:02.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:02.219+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.220745+0000) 2024-02-19T20:30:02.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:02.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:02.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:02.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:02.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:02.557+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:02.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:02.558+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:02.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:02.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:02.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:02.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:02.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:02.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:02.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:02.692+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:02.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:02.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:02.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:02.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:02.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:02.586+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:02.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:02.587+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:03.095 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:02.797+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.798752+0000) 2024-02-19T20:30:03.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:02.991+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:02.992018+0000) 2024-02-19T20:30:03.353 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:03.244+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64906 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:03.353 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:03.240+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.241630+0000) 2024-02-19T20:30:03.354 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:03.221+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.222751+0000) 2024-02-19T20:30:03.354 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:03.240+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.241896+0000) 2024-02-19T20:30:03.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:03.241+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.242203+0000) 2024-02-19T20:30:03.617 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:03.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:03.618 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:03.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:03.618 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:03.538+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:03.618 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:03.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:03.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:03.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:03.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:03.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:03.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:03.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:03.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:03.626+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:03.963 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:03.797+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.798988+0000) 2024-02-19T20:30:03.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:03.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:03.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:03.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:03.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:03.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:03.964 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:03.697+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:04.213 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:03.991+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:03.992226+0000) 2024-02-19T20:30:04.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:04.221+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.222954+0000) 2024-02-19T20:30:04.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:04.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:04.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:04.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:04.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:04.549+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:04.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:04.550+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:04.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:04.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:04.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:04.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:04.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:04.668+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:04.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:04.669+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:04.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:04.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:04.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:04.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:04.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:04.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:04.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:04.686+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:05.095 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:04.801+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.802611+0000) 2024-02-19T20:30:05.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:04.994+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:04.995561+0000) 2024-02-19T20:30:05.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:05.224+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.225513+0000) 2024-02-19T20:30:05.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:05.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:05.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:05.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:05.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:05.639+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:05.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:05.640+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:05.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:05.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:05.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:05.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:05.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:05.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:05.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:05.591+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:05.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:05.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:05.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:05.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:05.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:05.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:05.784 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:05.701+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:06.095 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:05.804+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.805234+0000) 2024-02-19T20:30:06.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:05.997+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:05.998817+0000) 2024-02-19T20:30:06.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:06.226+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.227507+0000) 2024-02-19T20:30:06.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:06.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:06.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:06.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:06.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:06.593+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:06.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:06.594+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:06.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:06.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:06.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:06.681+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:06.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:06.681+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:06.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:06.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:06.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:06.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:06.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:06.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:06.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:06.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:07.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:07.000+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.001682+0000) 2024-02-19T20:30:07.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:06.807+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:06.808899+0000) 2024-02-19T20:30:07.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:07.228+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.229592+0000) 2024-02-19T20:30:07.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:07.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:07.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:07.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:07.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:07.683+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:07.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:07.684+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:07.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:07.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:07.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:07.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:07.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:07.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:07.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:07.687+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:07.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:07.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:07.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:07.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:07.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:07.560+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:07.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:07.561+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:08.095 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:07.810+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:07.811349+0000) 2024-02-19T20:30:08.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:08.002+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.003724+0000) 2024-02-19T20:30:08.363 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:08.252+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64945 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:08.364 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:08.248+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.249979+0000) 2024-02-19T20:30:08.364 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:08.248+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.249656+0000) 2024-02-19T20:30:08.364 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:08.231+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.232047+0000) 2024-02-19T20:30:08.365 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:08.249+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.250333+0000) 2024-02-19T20:30:08.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:08.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:08.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:08.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:08.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:08.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:08.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:08.586+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:08.969 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:08.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:08.969 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:08.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:08.970 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:08.691+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:08.970 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:08.692+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:08.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:08.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:08.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:08.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:08.970 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:08.702+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:08.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:08.702+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:08.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:08.810+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:08.811573+0000) 2024-02-19T20:30:09.222 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:09.002+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.003962+0000) 2024-02-19T20:30:09.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:09.231+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.232231+0000) 2024-02-19T20:30:09.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:09.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:09.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:09.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:09.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:09.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:09.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:09.621+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:09.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:09.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:09.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:09.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:09.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:09.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:09.745 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:09.644+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:09.996 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:09.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:09.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:09.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:09.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:09.752+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:09.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:09.752+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:09.997 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:09.812+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:09.813776+0000) 2024-02-19T20:30:10.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:10.005+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.006121+0000) 2024-02-19T20:30:10.252 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:10.233+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.234841+0000) 2024-02-19T20:30:10.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:10.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:10.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:10.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:10.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:10.598+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:10.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:10.599+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:10.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:10.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:10.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:10.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:10.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:10.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:10.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:10.634+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:11.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:11.007+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.008292+0000) 2024-02-19T20:30:11.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:10.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:11.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:10.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:11.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:10.797+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:11.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:10.798+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:11.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:10.816+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:10.817263+0000) 2024-02-19T20:30:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:11.235+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.236845+0000) 2024-02-19T20:30:11.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:11.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:11.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:11.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:11.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:11.605+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:11.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:11.606+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:11.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:11.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:11.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:11.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:11.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:11.628+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:11.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:11.628+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:12.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:12.010+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.011762+0000) 2024-02-19T20:30:12.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:11.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:12.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:11.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:12.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:11.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:12.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:11.794+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:12.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:11.819+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:11.820762+0000) 2024-02-19T20:30:12.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:12.237+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.238948+0000) 2024-02-19T20:30:12.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:12.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:12.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:12.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:12.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:12.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:12.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:12.630+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:12.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:12.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:12.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:12.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:12.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:12.570+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:12.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:12.571+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:13.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:13.014+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.015125+0000) 2024-02-19T20:30:13.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:12.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:13.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:12.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:13.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:12.784+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:13.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:12.785+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:13.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:12.823+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:12.824167+0000) 2024-02-19T20:30:13.369 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:13.259+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 64985 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:13.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:13.256+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.257232+0000) 2024-02-19T20:30:13.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:13.256+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.257330+0000) 2024-02-19T20:30:13.370 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:13.239+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.240966+0000) 2024-02-19T20:30:13.370 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:13.256+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.257727+0000) 2024-02-19T20:30:13.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:13.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:13.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:13.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:13.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:13.594+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:13.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:13.594+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:13.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:13.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:13.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:13.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:13.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:13.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:13.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:13.533+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:14.032 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:14.014+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.015347+0000) 2024-02-19T20:30:14.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:13.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:14.032 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:13.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:14.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:13.774+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:14.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:13.775+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:14.033 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:13.823+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:13.824394+0000) 2024-02-19T20:30:14.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:14.240+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.241156+0000) 2024-02-19T20:30:14.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:14.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:14.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:14.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:14.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:14.547+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:14.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:14.548+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:14.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:14.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:14.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:14.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:14.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:14.567+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:14.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:14.568+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:15.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:15.017+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.018817+0000) 2024-02-19T20:30:15.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:14.819+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:15.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:14.819+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:15.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:14.819+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:15.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:14.819+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:15.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:14.825+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:14.826567+0000) 2024-02-19T20:30:15.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:15.242+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.243165+0000) 2024-02-19T20:30:15.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:15.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:15.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:15.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:15.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:15.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:15.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:15.594+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:15.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:15.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:15.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:15.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:15.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:15.614+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:15.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:15.615+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:16.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:16.021+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.022333+0000) 2024-02-19T20:30:16.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:15.827+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:15.828704+0000) 2024-02-19T20:30:16.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:15.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:16.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:15.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:16.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:15.839+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:16.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:15.840+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:16.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:16.244+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.245172+0000) 2024-02-19T20:30:16.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:16.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:16.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:16.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:16.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:16.593+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:16.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:16.594+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:16.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:16.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:16.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:16.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:16.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:16.570+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:16.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:16.571+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:17.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:17.024+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.025771+0000) 2024-02-19T20:30:17.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:16.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:17.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:16.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:17.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:16.793+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:17.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:16.793+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:17.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:16.829+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:16.830903+0000) 2024-02-19T20:30:17.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:17.246+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.247145+0000) 2024-02-19T20:30:17.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:17.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:17.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:17.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:17.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:17.632+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:17.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:17.633+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:17.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:17.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:17.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:17.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:17.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:17.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:17.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:17.575+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:18.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:18.028+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.029214+0000) 2024-02-19T20:30:18.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:17.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:18.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:17.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:18.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:17.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:18.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:17.831+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:18.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:17.832+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:17.833500+0000) 2024-02-19T20:30:18.376 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:18.267+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65024 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:18.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:18.264+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.265977+0000) 2024-02-19T20:30:18.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:18.263+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.264750+0000) 2024-02-19T20:30:18.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:18.248+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.249128+0000) 2024-02-19T20:30:18.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:18.263+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.264688+0000) 2024-02-19T20:30:18.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:18.594+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:18.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:18.594+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:18.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:18.594+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:18.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:18.595+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:18.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:18.555+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:18.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:18.555+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:18.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:18.555+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:18.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:18.556+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:19.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:19.028+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:19.029411+0000) 2024-02-19T20:30:19.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:18.832+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:18.833720+0000) 2024-02-19T20:30:19.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:18.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:19.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:18.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:19.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:18.867+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:19.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:18.868+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:19.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:19.248+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:19.249327+0000) 2024-02-19T20:30:19.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:19.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:19.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:19.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:19.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:19.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:19.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:19.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:19.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:19.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:19.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:19.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:19.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:19.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:19.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:19.572+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:20.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:20.030+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.031385+0000) 2024-02-19T20:30:20.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:19.834+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:19.835766+0000) 2024-02-19T20:30:20.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:19.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:20.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:19.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:20.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:19.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:20.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:19.909+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:20.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:20.251+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.252739+0000) 2024-02-19T20:30:20.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:20.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:20.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:20.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:20.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:20.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:20.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:20.537+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:20.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:20.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:20.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:20.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:20.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:20.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:20.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:20.571+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:21.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:21.035+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:21.035097+0000) 2024-02-19T20:30:21.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:20.836+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:20.837894+0000) 2024-02-19T20:30:21.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:20.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:21.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:20.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:21.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:20.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:21.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:20.897+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:21.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:21.256+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:21.256363+0000) 2024-02-19T20:30:21.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:21.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:21.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:21.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:21.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:21.539+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:21.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:21.539+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:21.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:21.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:21.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:21.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:21.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:21.615+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:21.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:21.616+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:22.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:22.037+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:22.038933+0000) 2024-02-19T20:30:22.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:21.841+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:21.841556+0000) 2024-02-19T20:30:22.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:21.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:22.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:21.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:22.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:21.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:22.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:21.888+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:22.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:22.258+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:22.259095+0000) 2024-02-19T20:30:22.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:22.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:22.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:22.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:22.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:22.542+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:22.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:22.543+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:22.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:22.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:22.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:22.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:22.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:22.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:22.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:22.657+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:23.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:23.040+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.041611+0000) 2024-02-19T20:30:23.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:22.842+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:22.843632+0000) 2024-02-19T20:30:23.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:22.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:23.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:22.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:23.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:22.847+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:23.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:22.848+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:23.387 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:23.276+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65064 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:23.388 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:23.260+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.261943+0000) 2024-02-19T20:30:23.388 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:23.271+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.272846+0000) 2024-02-19T20:30:23.388 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:23.272+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.273245+0000) 2024-02-19T20:30:23.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:23.272+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.273363+0000) 2024-02-19T20:30:23.657 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:23.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:23.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:23.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:23.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:23.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:23.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:23.506+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:23.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:23.666+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:23.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:23.666+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:23.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:23.666+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:23.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:23.667+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:23.983 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:23.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:23.983 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:23.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:23.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:23.816+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:23.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:23.817+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:23.984 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:23.842+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:23.843815+0000) 2024-02-19T20:30:24.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:24.040+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:24.041845+0000) 2024-02-19T20:30:24.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:24.261+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:24.262182+0000) 2024-02-19T20:30:24.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:24.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:24.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:24.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:24.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:24.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:24.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:24.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:24.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:24.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:24.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:24.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:24.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:24.708+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:24.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:24.709+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:25.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:25.042+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.043811+0000) 2024-02-19T20:30:25.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:24.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:25.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:24.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:25.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:24.841+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:25.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:24.841+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:25.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:24.845+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:24.846342+0000) 2024-02-19T20:30:25.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:25.263+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.264658+0000) 2024-02-19T20:30:25.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:25.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:25.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:25.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:25.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:25.571+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:25.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:25.571+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:25.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:25.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:25.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:25.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:25.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:25.686+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:25.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:25.687+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:26.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:26.045+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:26.046250+0000) 2024-02-19T20:30:26.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:25.847+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:25.848814+0000) 2024-02-19T20:30:26.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:25.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:26.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:25.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:26.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:25.872+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:26.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:25.873+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:26.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:26.265+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:26.266730+0000) 2024-02-19T20:30:26.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:26.713+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:26.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:26.713+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:26.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:26.713+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:26.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:26.713+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:26.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:26.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:26.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:26.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:26.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:26.574+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:26.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:26.575+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:27.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:27.047+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:27.048733+0000) 2024-02-19T20:30:27.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:26.850+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:26.851329+0000) 2024-02-19T20:30:27.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:26.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:27.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:26.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:27.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:26.856+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:27.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:26.856+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:27.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:27.268+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:27.269062+0000) 2024-02-19T20:30:27.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:27.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:27.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:27.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:27.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:27.589+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:27.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:27.589+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:27.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:27.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:27.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:27.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:27.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:27.692+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:27.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:27.693+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:28.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:28.049+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.050892+0000) 2024-02-19T20:30:28.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:27.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:28.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:27.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:28.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:27.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:28.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:27.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:28.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:27.852+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:27.853501+0000) 2024-02-19T20:30:28.393 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:28.284+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65103 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:28.394 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:28.280+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.281853+0000) 2024-02-19T20:30:28.394 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:28.281+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.282794+0000) 2024-02-19T20:30:28.394 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:28.270+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.271119+0000) 2024-02-19T20:30:28.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:28.280+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.281255+0000) 2024-02-19T20:30:28.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:28.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:28.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:28.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:28.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:28.578+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:28.664 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:28.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:28.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:28.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:28.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:28.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:28.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:28.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:28.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:28.673+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:28.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:28.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:28.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:28.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:28.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:28.789+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:28.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:28.789+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:28.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:28.852+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:28.853745+0000) 2024-02-19T20:30:29.261 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:29.050+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:29.051099+0000) 2024-02-19T20:30:29.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:29.270+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:29.271421+0000) 2024-02-19T20:30:29.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:29.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:29.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:29.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:29.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:29.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:29.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:29.648+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:29.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:29.776+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:29.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:29.776+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:29.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:29.776+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:29.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:29.776+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:29.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:29.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:29.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:29.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:29.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:29.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:29.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:29.568+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:30.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:30.053+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.054620+0000) 2024-02-19T20:30:30.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:29.854+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:29.855855+0000) 2024-02-19T20:30:30.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:30.273+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.274999+0000) 2024-02-19T20:30:30.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:30.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:30.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:30.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:30.514 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:30.522+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:30.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:30.523+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:30.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:30.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:30.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:30.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:30.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:30.674+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:30.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:30.675+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:31.097 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:31.056+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:31.057227+0000) 2024-02-19T20:30:31.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:30.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:31.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:30.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:31.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:30.818+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:31.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:30.818+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:31.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:30.856+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:30.858004+0000) 2024-02-19T20:30:31.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:31.276+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:31.277192+0000) 2024-02-19T20:30:31.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:31.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:31.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:31.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:31.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:31.491+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:31.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:31.493+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:31.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:31.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:31.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:31.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:31.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:31.626+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:31.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:31.626+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:32.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:31.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:32.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:31.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:32.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:31.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:32.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:31.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:32.050 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:31.859+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:31.860088+0000) 2024-02-19T20:30:32.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:32.278+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:32.279248+0000) 2024-02-19T20:30:32.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:32.058+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:32.059716+0000) 2024-02-19T20:30:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:32.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:32.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:32.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:32.450+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:32.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:32.665+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:32.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:32.665+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:32.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:32.665+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:32.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:32.666+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:33.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:32.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:33.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:32.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:33.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:32.813+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:33.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:32.813+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:33.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:32.861+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:32.862692+0000) 2024-02-19T20:30:33.402 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:33.292+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65144 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:33.402 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:33.289+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.290340+0000) 2024-02-19T20:30:33.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:33.280+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.281293+0000) 2024-02-19T20:30:33.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:33.289+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.290474+0000) 2024-02-19T20:30:33.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:33.061+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.062264+0000) 2024-02-19T20:30:33.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:33.288+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.289990+0000) 2024-02-19T20:30:33.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:33.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:33.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:33.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:33.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:33.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:33.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:33.449+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:33.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:33.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:33.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:33.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:33.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:33.647+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:33.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:33.648+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:33.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:33.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:33.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:33.814+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:34.053 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:33.861+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:33.862918+0000) 2024-02-19T20:30:34.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:34.061+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:34.062590+0000) 2024-02-19T20:30:34.437 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:34.280+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:34.281508+0000) 2024-02-19T20:30:34.437 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:34.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:34.437 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:34.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:34.437 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:34.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:34.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:34.446+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:34.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:34.664+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:34.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:34.664+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:34.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:34.664+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:34.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:34.665+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:35.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:34.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:35.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:34.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:35.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:34.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:35.056 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:34.857+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:35.057 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:34.863+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:34.864831+0000) 2024-02-19T20:30:35.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:35.063+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.064803+0000) 2024-02-19T20:30:35.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:35.283+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.284828+0000) 2024-02-19T20:30:35.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:35.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:35.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:35.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:35.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:35.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:35.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:35.426+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:35.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:35.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:35.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:35.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:35.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:35.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:35.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:35.618+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:36.057 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:35.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:36.057 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:35.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:36.057 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:35.838+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:36.057 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:35.838+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:36.057 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:35.866+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:35.867078+0000) 2024-02-19T20:30:36.438 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:36.285+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:36.286865+0000) 2024-02-19T20:30:36.438 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:36.066+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:36.067000+0000) 2024-02-19T20:30:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:36.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:36.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:36.447+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:36.447+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:36.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:36.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:36.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:36.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:36.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:36.643+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:36.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:36.644+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:37.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:36.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:37.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:36.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:37.059 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:36.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:37.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:36.845+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:37.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:36.868+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:36.869235+0000) 2024-02-19T20:30:37.444 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:37.068+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:37.069206+0000) 2024-02-19T20:30:37.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:37.287+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:37.288977+0000) 2024-02-19T20:30:37.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:37.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:37.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:37.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:37.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:37.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:37.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:37.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:37.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:37.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:37.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:37.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:37.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:37.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:37.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:37.668+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:38.061 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:37.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:38.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:37.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:38.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:37.835+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:38.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:37.836+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:38.062 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:37.870+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:37.871575+0000) 2024-02-19T20:30:38.410 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:38.300+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65183 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:38.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:38.297+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.298282+0000) 2024-02-19T20:30:38.411 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:38.291+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.292568+0000) 2024-02-19T20:30:38.412 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:38.297+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.298706+0000) 2024-02-19T20:30:38.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:38.070+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.071382+0000) 2024-02-19T20:30:38.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:38.297+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.298158+0000) 2024-02-19T20:30:38.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:38.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:38.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:38.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:38.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:38.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:38.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:38.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:38.995 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:38.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:38.995 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:38.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:38.995 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:38.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:38.995 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:38.790+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:38.996 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:38.870+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:38.871783+0000) 2024-02-19T20:30:38.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:38.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:38.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:38.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:38.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:38.717+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:38.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:38.718+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:39.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:39.070+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:39.071610+0000) 2024-02-19T20:30:39.580 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:39.291+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:39.292785+0000) 2024-02-19T20:30:39.580 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:39.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:39.580 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:39.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:39.580 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:39.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:39.580 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:39.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:39.864 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:39.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:39.864 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:39.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:39.864 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:39.821+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:39.865 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:39.821+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:39.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:39.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:39.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:39.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:39.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:39.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:39.865 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:39.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:40.254 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:40.072+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.073622+0000) 2024-02-19T20:30:40.255 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:39.872+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:39.873989+0000) 2024-02-19T20:30:40.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:40.293+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.294889+0000) 2024-02-19T20:30:40.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:40.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:40.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:40.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:40.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:40.487+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:40.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:40.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:40.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:40.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:40.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:40.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:40.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:40.676+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:40.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:40.677+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:41.066 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:40.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:41.066 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:40.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:41.066 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:40.822+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:41.066 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:40.822+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:41.066 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:40.874+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:40.876004+0000) 2024-02-19T20:30:41.490 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:41.295+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:41.296930+0000) 2024-02-19T20:30:41.490 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:41.074+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:41.075726+0000) 2024-02-19T20:30:41.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:41.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:41.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:41.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:41.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:41.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:41.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:41.500+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:41.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:41.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:41.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:41.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:41.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:41.663+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:41.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:41.663+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:42.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:41.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:42.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:41.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:42.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:41.866+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:42.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:41.867+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:42.068 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:41.877+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:41.878240+0000) 2024-02-19T20:30:42.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:42.298+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:42.299055+0000) 2024-02-19T20:30:42.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:42.076+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:42.077932+0000) 2024-02-19T20:30:42.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:42.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:42.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:42.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:42.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:42.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:42.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:42.476+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:42.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:42.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:42.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:42.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:42.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:42.657+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:42.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:42.657+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:43.071 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:42.879+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:42.880513+0000) 2024-02-19T20:30:43.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:42.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:43.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:42.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:43.072 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:42.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:43.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:42.888+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:43.416 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:43.308+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65223 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:43.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:43.305+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.306362+0000) 2024-02-19T20:30:43.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:43.300+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.301152+0000) 2024-02-19T20:30:43.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:43.305+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.307011+0000) 2024-02-19T20:30:43.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:43.079+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.080252+0000) 2024-02-19T20:30:43.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:43.305+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.306519+0000) 2024-02-19T20:30:43.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:43.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:43.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:43.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:43.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:43.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:43.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:43.520+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:43.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:43.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:43.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:43.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:43.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:43.635+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:43.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:43.636+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:44.070 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:43.879+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:43.880745+0000) 2024-02-19T20:30:44.070 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:43.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:44.071 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:43.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:44.071 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:43.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:44.071 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:43.898+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:44.503 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:44.079+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:44.080463+0000) 2024-02-19T20:30:44.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:44.300+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:44.301342+0000) 2024-02-19T20:30:44.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:44.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:44.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:44.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:44.504 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:44.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:44.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:44.513+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:44.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:44.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:44.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:44.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:44.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:44.668+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:44.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:44.669+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:45.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:44.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:45.074 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:44.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:45.074 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:44.877+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:45.074 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:44.878+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:45.074 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:44.881+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:44.882847+0000) 2024-02-19T20:30:45.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:45.302+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.303524+0000) 2024-02-19T20:30:45.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:45.082+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.083666+0000) 2024-02-19T20:30:45.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:45.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:45.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:45.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:45.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:45.544+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:45.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:45.545+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:45.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:45.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:45.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:45.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:45.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:45.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:45.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:45.704+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:46.077 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:45.885+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:45.886442+0000) 2024-02-19T20:30:46.077 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:45.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:46.077 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:45.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:46.077 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:45.911+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:46.077 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:45.911+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:46.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:46.306+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:46.307121+0000) 2024-02-19T20:30:46.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:46.085+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:46.086915+0000) 2024-02-19T20:30:46.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:46.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:46.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:46.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:46.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:46.558+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:46.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:46.559+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:46.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:46.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:46.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:46.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:46.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:46.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:46.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:46.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:47.080 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:46.889+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:46.890093+0000) 2024-02-19T20:30:47.080 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:46.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:47.080 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:46.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:47.081 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:46.949+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:47.081 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:46.950+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:47.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:47.308+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:47.309158+0000) 2024-02-19T20:30:47.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:47.089+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:47.090304+0000) 2024-02-19T20:30:47.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:47.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:47.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:47.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:47.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:47.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:47.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:47.555+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:47.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:47.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:47.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:47.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:47.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:47.722+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:47.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:47.723+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:48.082 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:47.891+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:47.892956+0000) 2024-02-19T20:30:48.083 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:47.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:48.083 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:47.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:48.083 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:47.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:48.083 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:47.985+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:48.083 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:48.091+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.092581+0000) 2024-02-19T20:30:48.426 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:48.317+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65262 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:48.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:48.313+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.314241+0000) 2024-02-19T20:30:48.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:48.313+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.314279+0000) 2024-02-19T20:30:48.427 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:48.313+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.314622+0000) 2024-02-19T20:30:48.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:48.313+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.314375+0000) 2024-02-19T20:30:48.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:48.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:48.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:48.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:48.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:48.534+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:48.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:48.534+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:49.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:48.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:49.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:48.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:49.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:48.760+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:49.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:48.761+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:49.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:48.892+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:48.893207+0000) 2024-02-19T20:30:49.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:48.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:49.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:48.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:49.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:48.999+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:49.011 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:48.999+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:49.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:49.091+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:49.092788+0000) 2024-02-19T20:30:49.580 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:49.316+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:49.317237+0000) 2024-02-19T20:30:49.581 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:49.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:49.581 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:49.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:49.581 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:49.553+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:49.581 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:49.554+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:49.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:49.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:49.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:49.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:49.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:49.809+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:49.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:49.810+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:50.256 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:50.093+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.094865+0000) 2024-02-19T20:30:50.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:49.894+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:49.895257+0000) 2024-02-19T20:30:50.256 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:49.980+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:50.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:49.980+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:50.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:49.980+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:50.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:49.980+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:50.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:50.318+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.319333+0000) 2024-02-19T20:30:50.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:50.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:50.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:50.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:50.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:50.559+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:50.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:50.560+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:51.087 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:50.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:51.087 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:50.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:51.087 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:50.848+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:51.087 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:50.848+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:51.087 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:50.896+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:50.897750+0000) 2024-02-19T20:30:51.088 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:50.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:51.088 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:50.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:51.088 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:50.979+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:51.088 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:50.980+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:51.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:51.095+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:51.096933+0000) 2024-02-19T20:30:51.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:51.320+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:51.321392+0000) 2024-02-19T20:30:51.517 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:51.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:51.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:51.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:51.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:51.525+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:51.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:51.526+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:52.090 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:51.898+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:51.899876+0000) 2024-02-19T20:30:52.091 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.025+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:52.091 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.025+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:52.091 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.025+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:52.091 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.025+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:52.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:51.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:52.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:51.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:52.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:51.831+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:52.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:51.832+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:52.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:52.323+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:52.324097+0000) 2024-02-19T20:30:52.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:52.099+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:52.100478+0000) 2024-02-19T20:30:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:52.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:52.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:52.548+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:52.549+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:53.092 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:52.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:53.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:52.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:53.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:52.824+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:53.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:52.825+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:53.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.901+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:52.902350+0000) 2024-02-19T20:30:53.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:53.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:53.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.991+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:53.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:52.992+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:53.436 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:53.325+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65302 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:53.437 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:53.321+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.322604+0000) 2024-02-19T20:30:53.437 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:53.320+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.321719+0000) 2024-02-19T20:30:53.437 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:53.323+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.324366+0000) 2024-02-19T20:30:53.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:53.101+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.102726+0000) 2024-02-19T20:30:53.438 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:53.321+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.322374+0000) 2024-02-19T20:30:53.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:53.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:53.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:53.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:53.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:53.554+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:53.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:53.555+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:54.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:53.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:54.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:53.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:54.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:53.835+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:54.093 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:53.836+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:54.093 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:53.901+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:53.902641+0000) 2024-02-19T20:30:54.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:53.955+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:54.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:53.955+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:54.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:53.955+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:54.094 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:53.956+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:54.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:54.325+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:54.326866+0000) 2024-02-19T20:30:54.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:54.101+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:54.102954+0000) 2024-02-19T20:30:54.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:54.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:54.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:54.537+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:54.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:54.538+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:55.095 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:54.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:55.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:54.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:55.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:54.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:55.096 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:54.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:55.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:54.903+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:54.904694+0000) 2024-02-19T20:30:55.096 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:54.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:55.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:54.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:55.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:54.969+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:55.097 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:54.969+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:55.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:55.104+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.105666+0000) 2024-02-19T20:30:55.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:55.327+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.328871+0000) 2024-02-19T20:30:55.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:55.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:55.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:55.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:55.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:55.568+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:55.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:55.568+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:56.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:55.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:56.098 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:55.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:56.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:55.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:56.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:55.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:56.099 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:55.905+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:55.906766+0000) 2024-02-19T20:30:56.100 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:55.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:56.100 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:55.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:56.100 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:55.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:56.100 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:55.954+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:56.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:56.107+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:56.108114+0000) 2024-02-19T20:30:56.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:56.331+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:56.332420+0000) 2024-02-19T20:30:56.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:56.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:56.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:56.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:56.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:56.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:56.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:56.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:57.100 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:56.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:57.100 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:56.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:57.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:56.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:57.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:56.892+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:57.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:56.907+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:56.908839+0000) 2024-02-19T20:30:57.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:56.923+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:57.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:56.923+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:57.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:56.923+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:57.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:56.923+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:57.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:57.109+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:57.110491+0000) 2024-02-19T20:30:57.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:57.335+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:57.336054+0000) 2024-02-19T20:30:57.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:57.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:57.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:57.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:57.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:57.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:57.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:57.623+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:58.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:57.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:58.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:57.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:58.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:57.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:58.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:57.882+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:58.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:57.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:58.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:57.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:58.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:57.886+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:58.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:57.886+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:58.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:57.909+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:57.910900+0000) 2024-02-19T20:30:58.443 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:30:58.333+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65341 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:30:58.444 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:58.329+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.330279+0000) 2024-02-19T20:30:58.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:58.329+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.331009+0000) 2024-02-19T20:30:58.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:58.335+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.336282+0000) 2024-02-19T20:30:58.444 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:58.111+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.112896+0000) 2024-02-19T20:30:58.445 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:58.329+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.330138+0000) 2024-02-19T20:30:58.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:58.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:58.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:58.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:58.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:58.646+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:58.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:58.647+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:59.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:58.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:59.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:58.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:59.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:58.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:30:59.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:58.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:30:59.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:58.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:59.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:58.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:59.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:58.860+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:30:59.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:58.861+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:30:59.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:58.910+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:58.911159+0000) 2024-02-19T20:30:59.489 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:59.338+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:59.339378+0000) 2024-02-19T20:30:59.489 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:59.112+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:59.113149+0000) 2024-02-19T20:30:59.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:59.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:59.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:59.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:59.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:59.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:30:59.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:30:59.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:00.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:59.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:00.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:59.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:00.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:59.971+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:00.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:30:59.971+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:00.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:59.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:00.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:59.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:00.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:59.812+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:00.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:59.812+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:00.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:30:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:30:59.912+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:30:59.913150+0000) 2024-02-19T20:31:00.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:00.340+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.341372+0000) 2024-02-19T20:31:00.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:00.115+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.116652+0000) 2024-02-19T20:31:00.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:00.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:00.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:00.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:00.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:00.661+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:00.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:00.662+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:01.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:00.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:01.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:00.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:01.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:00.850+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:01.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:00.850+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:01.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:00.914+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:00.915137+0000) 2024-02-19T20:31:01.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:00.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:01.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:00.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:01.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:00.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:01.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:00.944+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:01.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:01.342+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:01.343595+0000) 2024-02-19T20:31:01.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:01.117+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:01.118984+0000) 2024-02-19T20:31:01.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:01.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:01.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:01.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:01.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:01.625+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:01.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:01.626+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:02.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:01.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:02.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:01.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:02.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:01.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:02.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:01.965+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:02.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:01.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:02.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:01.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:02.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:01.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:02.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:01.885+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:02.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:01.917+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:01.918055+0000) 2024-02-19T20:31:02.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:02.344+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:02.345816+0000) 2024-02-19T20:31:02.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:02.120+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:02.121055+0000) 2024-02-19T20:31:02.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:02.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:02.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:02.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:02.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:02.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:02.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:02.586+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:03.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:02.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:03.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:02.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:03.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:02.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:03.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:02.959+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:03.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:02.920+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:02.921716+0000) 2024-02-19T20:31:03.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:02.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:03.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:02.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:03.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:02.927+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:03.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:02.927+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:03.450 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:03.342+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65381 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:03.450 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:03.336+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.337934+0000) 2024-02-19T20:31:03.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:03.337+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.338331+0000) 2024-02-19T20:31:03.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:03.345+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.346052+0000) 2024-02-19T20:31:03.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:03.123+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.124395+0000) 2024-02-19T20:31:03.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:03.337+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.338537+0000) 2024-02-19T20:31:03.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:03.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:03.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:03.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:03.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:03.595+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:03.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:03.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:04.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:04.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:04.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:04.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:04.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:04.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:04.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:04.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:04.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:03.921+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:03.922051+0000) 2024-02-19T20:31:04.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:03.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:04.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:03.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:04.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:03.968+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:04.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:03.968+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:04.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:04.123+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:04.124716+0000) 2024-02-19T20:31:04.532 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:04.347+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:04.348578+0000) 2024-02-19T20:31:04.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:04.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:04.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:04.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:04.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:04.630+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:04.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:04.630+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:05.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:05.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:05.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:05.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.029+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:05.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:04.923+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:04.924058+0000) 2024-02-19T20:31:05.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:04.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:05.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:04.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:05.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:04.976+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:05.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:04.976+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:05.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:05.350+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.351153+0000) 2024-02-19T20:31:05.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.127+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.128266+0000) 2024-02-19T20:31:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:05.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:05.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:05.636+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:05.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:05.637+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:06.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:05.925+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:05.926067+0000) 2024-02-19T20:31:06.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:05.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:06.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:05.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:06.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:05.990+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:06.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:05.991+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:06.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:06.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:06.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:06.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:05.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:06.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:06.352+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:06.353749+0000) 2024-02-19T20:31:06.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:06.129+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:06.130383+0000) 2024-02-19T20:31:06.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:06.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:06.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:06.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:06.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:06.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:06.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:06.593+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:07.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:06.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:07.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:06.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:07.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:06.961+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:07.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:06.962+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:07.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:06.927+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:06.928120+0000) 2024-02-19T20:31:07.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:07.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:07.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:07.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:07.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:07.001+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:07.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:07.002+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:07.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:07.356+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:07.356193+0000) 2024-02-19T20:31:07.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:07.132+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:07.132497+0000) 2024-02-19T20:31:07.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:07.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:07.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:07.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:07.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:07.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:07.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:07.623+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:08.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:07.931+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:07.931830+0000) 2024-02-19T20:31:08.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:08.042+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:08.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:08.042+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:08.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:08.042+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:08.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:08.043+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:08.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:07.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:08.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:07.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:08.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:07.997+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:08.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:07.998+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:08.357 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:08.350+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65420 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:08.358 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:08.347+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.347573+0000) 2024-02-19T20:31:08.358 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:08.347+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.347269+0000) 2024-02-19T20:31:08.358 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:08.356+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.356531+0000) 2024-02-19T20:31:08.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:08.134+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.134733+0000) 2024-02-19T20:31:08.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:08.348+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.348414+0000) 2024-02-19T20:31:08.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:08.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:08.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:08.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:08.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:08.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:08.922 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:08.619+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:09.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:09.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:09.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:09.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:09.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:09.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:09.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:09.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:09.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:09.134+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:09.135016+0000) 2024-02-19T20:31:09.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:08.931+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:08.932157+0000) 2024-02-19T20:31:09.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:09.046+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:09.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:09.046+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:09.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:09.046+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:09.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:09.046+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:09.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:09.358+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:09.359203+0000) 2024-02-19T20:31:09.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:09.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:09.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:09.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:09.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:09.581+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:09.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:09.582+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:10.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:10.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:10.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:10.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:10.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:10.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:10.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:10.017+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:10.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:09.934+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:09.935494+0000) 2024-02-19T20:31:10.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:10.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:10.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:10.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:10.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:10.011+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:10.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:10.011+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:10.360+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.361809+0000) 2024-02-19T20:31:10.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:10.136+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.137237+0000) 2024-02-19T20:31:10.801 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:10.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:10.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:10.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:10.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:10.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:10.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:10.632+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:11.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:11.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:11.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:11.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:11.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:11.045+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:11.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:11.045+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:11.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:10.936+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:10.937577+0000) 2024-02-19T20:31:11.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:11.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:11.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:11.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:11.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:11.016+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:11.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:11.016+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:11.353 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:11.138+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:11.139310+0000) 2024-02-19T20:31:11.632 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:11.362+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:11.363947+0000) 2024-02-19T20:31:11.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:11.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:11.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:11.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:11.929 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:11.642+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:11.930 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:11.643+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:12.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:12.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:12.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:12.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:12.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:12.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:12.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:12.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:12.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:12.141+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:12.142615+0000) 2024-02-19T20:31:12.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:11.938+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:11.939660+0000) 2024-02-19T20:31:12.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:12.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:12.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:12.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:12.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:12.027+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:12.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:12.027+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:12.681 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:12.365+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:12.366204+0000) 2024-02-19T20:31:12.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:12.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:12.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:12.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:12.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:12.690+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:12.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:12.691+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:13.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:13.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:13.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:13.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:13.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:13.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:13.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:13.068+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:13.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:13.144+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.145067+0000) 2024-02-19T20:31:13.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:12.941+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:12.942171+0000) 2024-02-19T20:31:13.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:13.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:13.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:13.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:13.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:13.043+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:13.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:13.043+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:13.640 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:13.357+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65459 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:13.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:13.354+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.355681+0000) 2024-02-19T20:31:13.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:13.354+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.355437+0000) 2024-02-19T20:31:13.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:13.353+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.354973+0000) 2024-02-19T20:31:13.642 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:13.365+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.366472+0000) 2024-02-19T20:31:13.642 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:13.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:13.642 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:13.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:13.642 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:13.650+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:13.932 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:13.650+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:14.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:14.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:14.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:14.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:14.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:14.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:14.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:14.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:14.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:14.144+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:14.145324+0000) 2024-02-19T20:31:14.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:13.941+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:13.942472+0000) 2024-02-19T20:31:14.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:14.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:14.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:14.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:14.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:14.060+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:14.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:14.060+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:14.676 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:14.369+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:14.370015+0000) 2024-02-19T20:31:14.676 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:14.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:14.676 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:14.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:14.676 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:14.685+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:14.934 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:14.686+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:15.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:15.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:15.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:15.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:15.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:15.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:15.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:15.145+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:15.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:15.146+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.147354+0000) 2024-02-19T20:31:15.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:14.943+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:14.944532+0000) 2024-02-19T20:31:15.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:15.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:15.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:15.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:15.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:15.091+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:15.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:15.092+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:15.648 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:15.370+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.371977+0000) 2024-02-19T20:31:15.936 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:15.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:15.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:15.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:15.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:15.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:15.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:15.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:16.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:16.149+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:16.151027+0000) 2024-02-19T20:31:16.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:16.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:16.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:16.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:16.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:16.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:16.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:16.194+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:16.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:15.945+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:15.946621+0000) 2024-02-19T20:31:16.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:16.110+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:16.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:16.110+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:16.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:16.110+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:16.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:16.111+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:16.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:16.374+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:16.375605+0000) 2024-02-19T20:31:16.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:16.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:16.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:16.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:16.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:16.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:16.939 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:16.694+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:17.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:17.151+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:17.153007+0000) 2024-02-19T20:31:17.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:17.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:17.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:17.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:17.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:17.168+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:17.282 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:17.168+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:17.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:16.948+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:16.949338+0000) 2024-02-19T20:31:17.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:17.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:17.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:17.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:17.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:17.109+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:17.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:17.109+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:17.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:17.376+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:17.377623+0000) 2024-02-19T20:31:17.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:17.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:17.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:17.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:17.654 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:17.663+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:17.941 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:17.664+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:18.204 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:18.154+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.155099+0000) 2024-02-19T20:31:18.204 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:18.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:18.205 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:18.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:18.205 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:18.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:18.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:17.950+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:17.951592+0000) 2024-02-19T20:31:18.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:18.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:18.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:18.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:18.205 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:18.153+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:18.206 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:18.153+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:18.475 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:18.365+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65499 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:18.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:18.361+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.362480+0000) 2024-02-19T20:31:18.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:18.362+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.363401+0000) 2024-02-19T20:31:18.476 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:18.376+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.377886+0000) 2024-02-19T20:31:18.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:18.214+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:18.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:18.362+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.363657+0000) 2024-02-19T20:31:18.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:18.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:18.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:18.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:18.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:18.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:18.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:18.713+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:19.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:18.950+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:18.951914+0000) 2024-02-19T20:31:19.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:19.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:19.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:19.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:19.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:19.146+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:19.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:19.146+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:19.370 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:19.154+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:19.155608+0000) 2024-02-19T20:31:19.370 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:19.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:19.370 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:19.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:19.370 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:19.213+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:19.371 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:19.214+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:19.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:19.378+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:19.379932+0000) 2024-02-19T20:31:19.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:19.743+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:19.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:19.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:19.735 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:19.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:19.999 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:19.744+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:20.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:19.952+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:19.953986+0000) 2024-02-19T20:31:20.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:20.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:20.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:20.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:20.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:20.115+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:20.257 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:20.116+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:20.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:20.156+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.157545+0000) 2024-02-19T20:31:20.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:20.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:20.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:20.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:20.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:20.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:20.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:20.224+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:20.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:20.381+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.382551+0000) 2024-02-19T20:31:20.802 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:20.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:20.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:20.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:20.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:20.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:20.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:20.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:21.103 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:20.955+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:20.956066+0000) 2024-02-19T20:31:21.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:21.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:21.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:21.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:21.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:21.147+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:21.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:21.147+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:21.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:21.158+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:21.159555+0000) 2024-02-19T20:31:21.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:21.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:21.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:21.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:21.375 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:21.269+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:21.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:21.269+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:21.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:21.384+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:21.385117+0000) 2024-02-19T20:31:21.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:21.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:21.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:21.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:21.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:21.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:21.998 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:21.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:21.998 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:21.957+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:21.958129+0000) 2024-02-19T20:31:22.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:22.160+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:22.161591+0000) 2024-02-19T20:31:22.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:22.123+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:22.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:22.123+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:22.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:22.123+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:22.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:22.124+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:22.583 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:22.387+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:22.388708+0000) 2024-02-19T20:31:22.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:22.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:22.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:22.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:22.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:22.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:22.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:22.300+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:22.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:22.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:22.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:22.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:22.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:22.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:22.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:22.726+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:23.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:23.163+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.164072+0000) 2024-02-19T20:31:23.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:22.960+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:22.961835+0000) 2024-02-19T20:31:23.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:23.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:23.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:23.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:23.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:23.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:23.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:23.169+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:23.584 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:23.373+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65538 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:23.584 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:23.370+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.371461+0000) 2024-02-19T20:31:23.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:23.369+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.370527+0000) 2024-02-19T20:31:23.585 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:23.388+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.389042+0000) 2024-02-19T20:31:23.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:23.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:23.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:23.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:23.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:23.304+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:23.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:23.305+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:23.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:23.370+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.371342+0000) 2024-02-19T20:31:23.951 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:23.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:23.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:23.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:23.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:23.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:23.952 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:23.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:24.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:24.163+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:24.164347+0000) 2024-02-19T20:31:24.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:23.961+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:23.962262+0000) 2024-02-19T20:31:24.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:24.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:24.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:24.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:24.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:24.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:24.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:24.186+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:24.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:24.390+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:24.391194+0000) 2024-02-19T20:31:24.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:24.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:24.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:24.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:24.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:24.315+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:24.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:24.316+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:24.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:24.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:24.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:24.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:24.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:24.712+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:24.954 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:24.713+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:25.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:25.166+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.167757+0000) 2024-02-19T20:31:25.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:24.963+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:24.964423+0000) 2024-02-19T20:31:25.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:25.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:25.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:25.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:25.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:25.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:25.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:25.178+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:25.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:25.392+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.393373+0000) 2024-02-19T20:31:25.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:25.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:25.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:25.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:25.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:25.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:25.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:25.328+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:25.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:25.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:25.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:25.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:25.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:25.704+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:25.956 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:25.705+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:26.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:26.168+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:26.169860+0000) 2024-02-19T20:31:26.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:25.965+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:25.966590+0000) 2024-02-19T20:31:26.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:26.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:26.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:26.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:26.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:26.143+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:26.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:26.144+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:26.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:26.395+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:26.396738+0000) 2024-02-19T20:31:26.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:26.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:26.584 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:26.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:26.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:26.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:26.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:26.309+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:26.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:26.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:26.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:26.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:26.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:26.669+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:26.958 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:26.670+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:27.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:27.171+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:27.172506+0000) 2024-02-19T20:31:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:26.967+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:26.968760+0000) 2024-02-19T20:31:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:27.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:27.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:27.163+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:27.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:27.164+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:27.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:27.397+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:27.398785+0000) 2024-02-19T20:31:27.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:27.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:27.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:27.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:27.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:27.332+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:27.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:27.333+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:27.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:27.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:27.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:27.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:27.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:27.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:27.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:27.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:28.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:28.173+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.174600+0000) 2024-02-19T20:31:28.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:27.969+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:27.971006+0000) 2024-02-19T20:31:28.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:28.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:28.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:28.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:28.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:28.211+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:28.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:28.211+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:28.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:28.383+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65578 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:28.649 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:28.378+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.379303+0000) 2024-02-19T20:31:28.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:28.378+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.379491+0000) 2024-02-19T20:31:28.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:28.398+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.399168+0000) 2024-02-19T20:31:28.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:28.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:28.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:28.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:28.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:28.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:28.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:28.362+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:28.650 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:28.377+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.378884+0000) 2024-02-19T20:31:28.960 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:28.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:28.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:28.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:28.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:28.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:28.961 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:28.658+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:29.250 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:29.173+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:29.174885+0000) 2024-02-19T20:31:29.250 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:28.970+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:28.971364+0000) 2024-02-19T20:31:29.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:29.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:29.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:29.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:29.251 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:29.259+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:29.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:29.260+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:29.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:29.401+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:29.402795+0000) 2024-02-19T20:31:29.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:29.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:29.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:29.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:29.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:29.383+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:29.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:29.384+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:29.803 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:29.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:29.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:29.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:29.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:29.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:29.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:29.654+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:30.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:29.972+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:29.973538+0000) 2024-02-19T20:31:30.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:30.176+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.177470+0000) 2024-02-19T20:31:30.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:30.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:30.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:30.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:30.376 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:30.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:30.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:30.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:30.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:30.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:30.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:30.229+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:30.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:30.230+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:30.658 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:30.386+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:30.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:30.405+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.406379+0000) 2024-02-19T20:31:30.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:30.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:30.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:30.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:30.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:30.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:30.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:30.668+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:31.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:31.178+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:31.179566+0000) 2024-02-19T20:31:31.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:30.974+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:30.975819+0000) 2024-02-19T20:31:31.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:31.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:31.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:31.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:31.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:31.184+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:31.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:31.184+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:31.667 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:31.407+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:31.408412+0000) 2024-02-19T20:31:31.667 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:31.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:31.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:31.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:31.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:31.350+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:31.668 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:31.350+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:31.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:31.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:31.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:31.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:31.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:31.676+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:31.967 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:31.677+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:32.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:32.180+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:32.181669+0000) 2024-02-19T20:31:32.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:31.977+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:31.978125+0000) 2024-02-19T20:31:32.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:32.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:32.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:32.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:32.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:32.138+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:32.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:32.139+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:32.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:32.409+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:32.410568+0000) 2024-02-19T20:31:32.679 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:32.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:32.679 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:32.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:32.679 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:32.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:32.679 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:32.382+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:32.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:32.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:32.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:32.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:32.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:32.707+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:32.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:32.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:33.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:33.182+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.183728+0000) 2024-02-19T20:31:33.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:32.979+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:32.980415+0000) 2024-02-19T20:31:33.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:33.164+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:33.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:33.164+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:33.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:33.164+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:33.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:33.164+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:33.670 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:33.391+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65617 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:33.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:33.388+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.389524+0000) 2024-02-19T20:31:33.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:33.387+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.388970+0000) 2024-02-19T20:31:33.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:33.409+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.410966+0000) 2024-02-19T20:31:33.671 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:33.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:33.671 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:33.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:33.671 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:33.379+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:33.672 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:33.380+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:33.672 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:33.387+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.388474+0000) 2024-02-19T20:31:33.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:33.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:33.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:33.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:33.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:33.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:33.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:33.680+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:34.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:34.183+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:34.184139+0000) 2024-02-19T20:31:34.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:33.979+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:33.980836+0000) 2024-02-19T20:31:34.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:34.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:34.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:34.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:34.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:34.122+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:34.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:34.123+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:34.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:34.412+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:34.413520+0000) 2024-02-19T20:31:34.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:34.675 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:34.675 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:34.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:34.675 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:34.390+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:34.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:34.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:34.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:34.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:34.972 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:34.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:34.973 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:34.685+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:35.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:35.185+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.186838+0000) 2024-02-19T20:31:35.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:34.981+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:34.982866+0000) 2024-02-19T20:31:35.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:35.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:35.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:35.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:35.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:35.165+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:35.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:35.165+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:35.637 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:35.414+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.415601+0000) 2024-02-19T20:31:35.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:35.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:35.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:35.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:35.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:35.382+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:35.638 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:35.383+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:35.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:35.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:35.974 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:35.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:35.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:35.647+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:35.975 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:35.648+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:36.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:36.189+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:36.190252+0000) 2024-02-19T20:31:36.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:35.984+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:35.985052+0000) 2024-02-19T20:31:36.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:36.117+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:36.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:36.117+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:36.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:36.117+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:36.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:36.118+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:36.642 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:36.417+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:36.418030+0000) 2024-02-19T20:31:36.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:36.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:36.642 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:36.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:36.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:36.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:36.643 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:36.373+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:36.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:36.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:36.976 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:36.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:36.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:36.651+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:36.977 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:36.652+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:37.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:37.191+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:37.192297+0000) 2024-02-19T20:31:37.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:36.986+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:36.987191+0000) 2024-02-19T20:31:37.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:37.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:37.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:37.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:37.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:37.075+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:37.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:37.076+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:37.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:37.420+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:37.421408+0000) 2024-02-19T20:31:37.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:37.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:37.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:37.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:37.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:37.402+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:37.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:37.403+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:37.978 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:37.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:37.979 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:37.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:37.979 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:37.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:37.979 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:37.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:38.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:38.194+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.195791+0000) 2024-02-19T20:31:38.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:37.988+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:37.989420+0000) 2024-02-19T20:31:38.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:38.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:38.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:38.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:38.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:38.094+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:38.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:38.094+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:38.535 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:38.400+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65657 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:38.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:38.396+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.397136+0000) 2024-02-19T20:31:38.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:38.394+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.395936+0000) 2024-02-19T20:31:38.536 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:38.420+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.421710+0000) 2024-02-19T20:31:38.536 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:38.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:38.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:38.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:38.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:38.359+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:38.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:38.360+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:38.537 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:38.395+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.396487+0000) 2024-02-19T20:31:38.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:38.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:38.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:38.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:38.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:38.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:38.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:38.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:39.060 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:38.988+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:38.989864+0000) 2024-02-19T20:31:39.368 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:39.195+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:39.196149+0000) 2024-02-19T20:31:39.368 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:39.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:39.368 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:39.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:39.368 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:39.074+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:39.368 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:39.075+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:39.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:39.422+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:39.423713+0000) 2024-02-19T20:31:39.728 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:39.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:39.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:39.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:39.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:39.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:39.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:39.377+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:39.729 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:39.378+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:39.982 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:39.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:39.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:39.737+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:39.983 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:39.738+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:40.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:40.198+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.199720+0000) 2024-02-19T20:31:40.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:39.992+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:39.993521+0000) 2024-02-19T20:31:40.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:40.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:40.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:40.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:40.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:40.114+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:40.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:40.114+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:40.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:40.424+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.425772+0000) 2024-02-19T20:31:40.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:40.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:40.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:40.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:40.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:40.368+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:40.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:40.369+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:40.804 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:40.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:40.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:40.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:40.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:40.765+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:40.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:40.766+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:41.064 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:40.994+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:40.995659+0000) 2024-02-19T20:31:41.396 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:41.200+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:41.201778+0000) 2024-02-19T20:31:41.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:41.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:41.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:41.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:41.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:41.154+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:41.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:41.155+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:41.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:41.426+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:41.427907+0000) 2024-02-19T20:31:41.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:41.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:41.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:41.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:41.768 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:41.405+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:41.768 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:41.406+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:42.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:41.996+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:41.997705+0000) 2024-02-19T20:31:42.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:41.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:42.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:41.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:42.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:41.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:42.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:41.777+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:42.381 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:42.202+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:42.203897+0000) 2024-02-19T20:31:42.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:42.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:42.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:42.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:42.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:42.162+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:42.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:42.163+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:42.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:42.430+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:42.431275+0000) 2024-02-19T20:31:42.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:42.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:42.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:42.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:42.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:42.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:42.739 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:42.392+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:43.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:42.998+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:42.999802+0000) 2024-02-19T20:31:43.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:42.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:43.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:42.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:43.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:42.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:43.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:42.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:43.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:43.206+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.207444+0000) 2024-02-19T20:31:43.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:43.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:43.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:43.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:43.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:43.178+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:43.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:43.179+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:43.744 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:43.407+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65695 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:43.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:43.404+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.405256+0000) 2024-02-19T20:31:43.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:43.404+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.405718+0000) 2024-02-19T20:31:43.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:43.430+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.431602+0000) 2024-02-19T20:31:43.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:43.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:43.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:43.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:43.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:43.366+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:43.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:43.366+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:43.746 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:43.404+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:43.405659+0000) 2024-02-19T20:31:44.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:43.999+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:44.000265+0000) 2024-02-19T20:31:44.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:43.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:44.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:43.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:44.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:43.754+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:44.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:43.755+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:44.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:44.206+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:44.207736+0000) 2024-02-19T20:31:44.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:44.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:44.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:44.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:44.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:44.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:44.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:44.169+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:44.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:44.432+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:44.433799+0000) 2024-02-19T20:31:44.586 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:44.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:44.586 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:44.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:44.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:44.324+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:44.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:44.325+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:44.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:44.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:44.992 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:44.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:44.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:44.715+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:44.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:44.716+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:45.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:45.209+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.210261+0000) 2024-02-19T20:31:45.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:45.001+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.002971+0000) 2024-02-19T20:31:45.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:45.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:45.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:45.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:45.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:45.177+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:45.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:45.178+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:45.588 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:45.434+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:45.435970+0000) 2024-02-19T20:31:45.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:45.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:45.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:45.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:45.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:45.308+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:45.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:45.309+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:45.994 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:45.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:45.995 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:45.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:45.995 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:45.702+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:45.995 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:45.703+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:46.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:46.212+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:46.213245+0000) 2024-02-19T20:31:46.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:46.004+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:46.005635+0000) 2024-02-19T20:31:46.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:46.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:46.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:46.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:46.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:46.191+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:46.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:46.192+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:46.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:46.437+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:46.438556+0000) 2024-02-19T20:31:46.586 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:46.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:46.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:46.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:46.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:46.327+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:46.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:46.328+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:46.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:46.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:46.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:46.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:46.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:46.733+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:46.997 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:46.734+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:47.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:47.214+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:47.215428+0000) 2024-02-19T20:31:47.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:47.006+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:47.007903+0000) 2024-02-19T20:31:47.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:47.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:47.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:47.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:47.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:47.242+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:47.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:47.242+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:47.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:47.439+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:47.440960+0000) 2024-02-19T20:31:47.586 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:47.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:47.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:47.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:47.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:47.300+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:47.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:47.301+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:48.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:47.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:48.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:47.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:48.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:47.767+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:48.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:47.768+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:48.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:48.216+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.217676+0000) 2024-02-19T20:31:48.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:48.009+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.010479+0000) 2024-02-19T20:31:48.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:48.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:48.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:48.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:48.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:48.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:48.282 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:48.245+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:48.586 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:48.416+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65727 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:48.587 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:48.412+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.413303+0000) 2024-02-19T20:31:48.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:48.412+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.413254+0000) 2024-02-19T20:31:48.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:48.440+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.441465+0000) 2024-02-19T20:31:48.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:48.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:48.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:48.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:48.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:48.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:48.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:48.345+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:48.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:48.412+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:48.413865+0000) 2024-02-19T20:31:49.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:48.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:49.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:48.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:49.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:48.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:49.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:48.772+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:49.260 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:49.216+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:49.217982+0000) 2024-02-19T20:31:49.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:49.009+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:49.010931+0000) 2024-02-19T20:31:49.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:49.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:49.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:49.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:49.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:49.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:49.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:49.443+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:49.444929+0000) 2024-02-19T20:31:49.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:49.270+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:49.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:49.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:49.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:49.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:49.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:49.381+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:49.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:49.381+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:49.805 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:49.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:49.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:49.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:49.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:49.732+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:49.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:49.733+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:50.074 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:50.012+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.013504+0000) 2024-02-19T20:31:50.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:50.219+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.220380+0000) 2024-02-19T20:31:50.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:50.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:50.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:50.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:50.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:50.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:50.390 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:50.307+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:50.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:50.446+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:50.447480+0000) 2024-02-19T20:31:50.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:50.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:50.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:50.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:50.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:50.399+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:50.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:50.400+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:51.000 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:50.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:51.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:50.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:51.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:50.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:51.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:50.725+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:51.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:51.015+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:51.016924+0000) 2024-02-19T20:31:51.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:51.221+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:51.222413+0000) 2024-02-19T20:31:51.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:51.449+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:51.450102+0000) 2024-02-19T20:31:51.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:51.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:51.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:51.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:51.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:51.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:51.764 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:51.349+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:51.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:51.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:51.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:51.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:51.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:51.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:51.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:51.407+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:52.031 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:52.019+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:52.020253+0000) 2024-02-19T20:31:52.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:51.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:52.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:51.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:52.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:51.772+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:52.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:51.773+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:52.330 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:52.223+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:52.224862+0000) 2024-02-19T20:31:52.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:52.451+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:52.452130+0000) 2024-02-19T20:31:52.587 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:52.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:52.587 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:52.339+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:52.587 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:52.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:52.587 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:52.340+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:52.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:52.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:52.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:52.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:52.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:52.415+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:52.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:52.416+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:53.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:52.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:53.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:52.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:53.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:52.790+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:53.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:52.792+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:53.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:53.226+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.227676+0000) 2024-02-19T20:31:53.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:53.022+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.023735+0000) 2024-02-19T20:31:53.587 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:53.424+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65766 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:53.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:53.421+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.422291+0000) 2024-02-19T20:31:53.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:53.452+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.452601+0000) 2024-02-19T20:31:53.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:53.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:53.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:53.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:53.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:53.305+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:53.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:53.306+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:53.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:53.421+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.422511+0000) 2024-02-19T20:31:53.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:53.421+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:53.422031+0000) 2024-02-19T20:31:53.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:53.453+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:53.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:53.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:53.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:53.454+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:53.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:53.454+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:54.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:53.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:54.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:53.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:54.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:53.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:54.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:53.766+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:54.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:54.228+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:54.228162+0000) 2024-02-19T20:31:54.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:54.024+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:54.024232+0000) 2024-02-19T20:31:54.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:54.455+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:54.455628+0000) 2024-02-19T20:31:54.587 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:54.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:54.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:54.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:54.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:54.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:54.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:54.307+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:54.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:54.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:54.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:54.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:54.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:54.498+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:54.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:54.498+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:55.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:54.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:55.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:54.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:55.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:54.759+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:55.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:54.760+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:55.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:55.229+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.230538+0000) 2024-02-19T20:31:55.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:55.025+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.026686+0000) 2024-02-19T20:31:55.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:55.456+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:55.457682+0000) 2024-02-19T20:31:55.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:55.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:55.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:55.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:55.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:55.775+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:55.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:55.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:55.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:55.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:55.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:55.521+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:55.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:55.522+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:55.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:55.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:55.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:55.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:55.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:55.350+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:55.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:55.350+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:56.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:55.776+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:56.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:56.233+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:56.234131+0000) 2024-02-19T20:31:56.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:56.027+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:56.028814+0000) 2024-02-19T20:31:56.587 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:56.458+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:56.459699+0000) 2024-02-19T20:31:56.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:56.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:56.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:56.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:56.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:56.306+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:56.588 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:56.306+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:56.588 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:56.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:56.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:56.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:56.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:56.518+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:56.589 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:56.518+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:57.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:56.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:57.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:56.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:57.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:56.792+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:57.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:56.793+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:57.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:57.235+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:57.236683+0000) 2024-02-19T20:31:57.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:57.029+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:57.030974+0000) 2024-02-19T20:31:57.537 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:57.461+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:57.462262+0000) 2024-02-19T20:31:57.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:57.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:57.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:57.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:57.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:57.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:57.538 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:57.512+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:57.538 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:57.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:57.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:57.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:57.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:57.297+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:57.539 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:57.297+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:58.082 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:58.033+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.034662+0000) 2024-02-19T20:31:58.082 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:57.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:58.083 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:57.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:58.083 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:57.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:58.083 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:57.820+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:58.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:58.238+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.239180+0000) 2024-02-19T20:31:58.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:58.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:58.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:58.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:58.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:58.322+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:58.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:58.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:58.768 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:31:58.432+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65806 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:31:58.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:58.430+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.431557+0000) 2024-02-19T20:31:58.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:58.429+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.430548+0000) 2024-02-19T20:31:58.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:58.461+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.462638+0000) 2024-02-19T20:31:58.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:58.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:58.769 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:58.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:58.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:58.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:58.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:58.429+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:58.430435+0000) 2024-02-19T20:31:58.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:58.466+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:58.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:58.466+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:58.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:58.466+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:58.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:58.466+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:59.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:58.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:31:59.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:59.238+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:59.239687+0000) 2024-02-19T20:31:59.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:59.034+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:59.035143+0000) 2024-02-19T20:31:59.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:59.463+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:31:59.464824+0000) 2024-02-19T20:31:59.740 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:59.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:59.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:59.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:31:59.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:59.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:59.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:59.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:59.741 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:59.424+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:31:59.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:31:59.425+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:31:59.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:59.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:59.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:59.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:59.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:59.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:31:59.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:31:59.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:00.001 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:59.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:00.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:31:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:31:59.751+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:00.258 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:00.037+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.038747+0000) 2024-02-19T20:32:00.259 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:00.240+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.241792+0000) 2024-02-19T20:32:00.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:00.465+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:00.466985+0000) 2024-02-19T20:32:00.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:00.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:00.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:00.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:00.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:00.414+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:00.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:00.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:00.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:00.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:00.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:00.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:00.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:00.408+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:00.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:00.408+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:00.806 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:00.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:00.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:00.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:00.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:00.777+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:00.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:00.778+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:01.087 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:01.041+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:01.042209+0000) 2024-02-19T20:32:01.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:01.242+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:01.243892+0000) 2024-02-19T20:32:01.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:01.469+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:01.470713+0000) 2024-02-19T20:32:01.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:01.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:01.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:01.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:01.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:01.431+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:01.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:01.431+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:01.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:01.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:01.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:01.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:01.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:01.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:01.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:01.432+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:02.087 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:02.043+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:02.044609+0000) 2024-02-19T20:32:02.088 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:01.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:02.088 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:01.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:02.088 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:01.804+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:02.088 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:01.805+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:02.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:02.246+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:02.247552+0000) 2024-02-19T20:32:02.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:02.471+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:02.472756+0000) 2024-02-19T20:32:02.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:02.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:02.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:02.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:02.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:02.387+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:02.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:02.387+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:02.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:02.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:02.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:02.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:02.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:02.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:02.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:02.397+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:03.088 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:03.045+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.046637+0000) 2024-02-19T20:32:03.088 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:02.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:03.088 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:02.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:03.088 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:02.823+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:03.089 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:02.824+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:03.338 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:03.248+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.249716+0000) 2024-02-19T20:32:03.768 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:03.440+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65845 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:03.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:03.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:03.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:03.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:03.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:03.348+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:03.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:03.349+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:03.769 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:03.437+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.438017+0000) 2024-02-19T20:32:03.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:03.437+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.438256+0000) 2024-02-19T20:32:03.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:03.472+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.473133+0000) 2024-02-19T20:32:03.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:03.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:03.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:03.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:03.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:03.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:03.770 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:03.437+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:03.438031+0000) 2024-02-19T20:32:03.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:03.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:03.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:03.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:03.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:03.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:03.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:03.442+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:04.032 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:03.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:04.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:04.046+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:04.047042+0000) 2024-02-19T20:32:04.381 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:04.249+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:04.250199+0000) 2024-02-19T20:32:04.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:04.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:04.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:04.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:04.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:04.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:04.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:04.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:04.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:04.474+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:04.475183+0000) 2024-02-19T20:32:04.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:04.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:04.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:04.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:04.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:04.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:04.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:04.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:04.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:04.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:04.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:04.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:04.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:04.457+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:05.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:04.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:05.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:05.048+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.049054+0000) 2024-02-19T20:32:05.281 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:05.251+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.252225+0000) 2024-02-19T20:32:05.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:05.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:05.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:05.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:05.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:05.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:05.751 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:05.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:05.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:05.477+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:05.478761+0000) 2024-02-19T20:32:05.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:05.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:05.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:05.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:05.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:05.760+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:05.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:05.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:05.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:05.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:05.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:05.404+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:05.752 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:05.405+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:06.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:05.761+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:06.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:06.253+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:06.254435+0000) 2024-02-19T20:32:06.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:06.050+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:06.051077+0000) 2024-02-19T20:32:06.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:06.480+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:06.481269+0000) 2024-02-19T20:32:06.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:06.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:06.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:06.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:06.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:06.714+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:06.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:06.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:06.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:06.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:06.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:06.393+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:06.706 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:06.394+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:06.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:06.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:06.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:06.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:06.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:06.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:06.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:06.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:07.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:06.715+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:07.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:07.255+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:07.256563+0000) 2024-02-19T20:32:07.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:07.052+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:07.053065+0000) 2024-02-19T20:32:07.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:07.482+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:07.483526+0000) 2024-02-19T20:32:07.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:07.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:07.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:07.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:07.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:07.436+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:07.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:07.437+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:07.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:07.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:07.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:07.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:07.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:07.480+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:07.727 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:07.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:08.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:07.735+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:08.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:07.735+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:08.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:07.735+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:08.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:07.736+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:08.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:08.257+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.258677+0000) 2024-02-19T20:32:08.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:08.055+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.056740+0000) 2024-02-19T20:32:08.712 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:08.447+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65885 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:08.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:08.444+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.445550+0000) 2024-02-19T20:32:08.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:08.483+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.484054+0000) 2024-02-19T20:32:08.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:08.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:08.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:08.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:08.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:08.722+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:08.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:08.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:08.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:08.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:08.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:08.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:08.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:08.439+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:08.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:08.444+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.445287+0000) 2024-02-19T20:32:08.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:08.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:08.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:08.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:08.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:08.397+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:08.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:08.398+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:08.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:08.444+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:08.445744+0000) 2024-02-19T20:32:09.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:08.723+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:09.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:09.258+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:09.259033+0000) 2024-02-19T20:32:09.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:09.056+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:09.057234+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:09.485+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:09.486108+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:09.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:09.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:09.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:09.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:09.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:09.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:09.438+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:09.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:09.439+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:09.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:09.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:09.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:09.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:09.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:09.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:09.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:09.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:10.030 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:09.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:10.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:09.769+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:10.031 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:09.770+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:10.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:10.260+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.261061+0000) 2024-02-19T20:32:10.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:10.058+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.059285+0000) 2024-02-19T20:32:10.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:10.488+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:10.489691+0000) 2024-02-19T20:32:10.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:10.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:10.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:10.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:10.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:10.448+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:10.737 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:10.449+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:10.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:10.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:10.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:10.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:10.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:10.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:10.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:10.431+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:11.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:10.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:11.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:10.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:11.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:10.747+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:11.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:10.747+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:11.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:11.262+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:11.263086+0000) 2024-02-19T20:32:11.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:11.060+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:11.061363+0000) 2024-02-19T20:32:11.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:11.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:11.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:11.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:11.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:11.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:11.711 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:11.384+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:11.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:11.491+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:11.492292+0000) 2024-02-19T20:32:11.712 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:11.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:11.712 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:11.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:11.712 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:11.721+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:11.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:11.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:11.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:11.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:11.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:11.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:11.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:11.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:12.002 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:11.721+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:12.253 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:12.063+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:12.064075+0000) 2024-02-19T20:32:12.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:12.493+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:12.494478+0000) 2024-02-19T20:32:12.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:12.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:12.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:12.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:12.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:12.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:12.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:12.512+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:12.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:12.264+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:12.265066+0000) 2024-02-19T20:32:12.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:12.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:12.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:12.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:12.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:12.420+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:12.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:12.421+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:12.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:12.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:12.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:12.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:12.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:12.710+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:12.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:12.711+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:13.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:13.065+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.066156+0000) 2024-02-19T20:32:13.438 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:13.266+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.267496+0000) 2024-02-19T20:32:13.780 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:13.455+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65924 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:13.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:13.452+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.453616+0000) 2024-02-19T20:32:13.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:13.493+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.494875+0000) 2024-02-19T20:32:13.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:13.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:13.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:13.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:13.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:13.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:13.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:13.695+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:13.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:13.451+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.452905+0000) 2024-02-19T20:32:13.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:13.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:13.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:13.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:13.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:13.468+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:13.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:13.468+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:13.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:13.452+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:13.453723+0000) 2024-02-19T20:32:13.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:13.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:13.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:13.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:13.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:13.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:13.783 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:13.512+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:14.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:14.065+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:14.066538+0000) 2024-02-19T20:32:14.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:14.266+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:14.267912+0000) 2024-02-19T20:32:14.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:14.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:14.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:14.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:14.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:14.547+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:14.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:14.548+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:14.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:14.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:14.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:14.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:14.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:14.514+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:14.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:14.515+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:14.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:14.495+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:14.496910+0000) 2024-02-19T20:32:14.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:14.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:14.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:14.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:14.784 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:14.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:14.785 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:14.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:15.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:15.067+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.068628+0000) 2024-02-19T20:32:15.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:15.269+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.270040+0000) 2024-02-19T20:32:15.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:15.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:15.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:15.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:15.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:15.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:15.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:15.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:15.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:15.499+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:15.500525+0000) 2024-02-19T20:32:15.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:15.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:15.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:15.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:15.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:15.697+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:15.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:15.698+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:15.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:15.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:15.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:15.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:15.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:15.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:15.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:15.477+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:16.474 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:16.069+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:16.070690+0000) 2024-02-19T20:32:16.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:16.271+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:16.272430+0000) 2024-02-19T20:32:16.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:16.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:16.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:16.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:16.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:16.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:16.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:16.485+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:16.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:16.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:16.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:16.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:16.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:16.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:16.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:16.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:16.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:16.501+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:16.502640+0000) 2024-02-19T20:32:16.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:16.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:16.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:16.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:16.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:16.706+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:16.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:16.707+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:17.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:17.273+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:17.274549+0000) 2024-02-19T20:32:17.464 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:17.071+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:17.072879+0000) 2024-02-19T20:32:17.714 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:17.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:17.714 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:17.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:17.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:17.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:17.715 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:17.475+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:17.715 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:17.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:17.715 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:17.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:17.715 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:17.511+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:17.715 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:17.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:17.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:17.503+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:17.504685+0000) 2024-02-19T20:32:17.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:17.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:17.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:17.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:17.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:17.724+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:18.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:17.725+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:18.266 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:18.074+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.075058+0000) 2024-02-19T20:32:18.530 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:18.464+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 65964 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:18.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:18.460+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.461542+0000) 2024-02-19T20:32:18.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:18.460+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.461868+0000) 2024-02-19T20:32:18.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:18.504+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.505244+0000) 2024-02-19T20:32:18.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:18.276+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.277064+0000) 2024-02-19T20:32:18.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:18.459+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:18.460630+0000) 2024-02-19T20:32:18.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:18.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:18.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:18.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:18.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:18.512+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:18.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:18.512+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:18.808 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:18.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:18.808 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:18.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:18.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:18.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:18.809 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:18.541+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:18.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:18.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:18.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:18.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:18.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:18.730+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:18.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:18.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:19.477 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:19.074+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:19.075589+0000) 2024-02-19T20:32:19.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:19.276+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:19.277602+0000) 2024-02-19T20:32:19.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:19.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:19.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:19.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:19.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:19.488+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:19.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:19.488+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:19.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:19.506+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:19.507376+0000) 2024-02-19T20:32:19.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:19.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:19.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:19.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:19.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:19.691+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:19.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:19.692+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:19.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:19.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:19.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:19.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:19.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:19.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:19.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:19.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:20.472 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:20.076+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.077652+0000) 2024-02-19T20:32:20.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:20.278+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.279617+0000) 2024-02-19T20:32:20.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:20.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:20.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:20.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:20.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:20.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:20.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:20.483+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:20.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:20.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:20.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:20.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:20.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:20.527+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:20.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:20.527+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:20.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:20.508+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:20.509535+0000) 2024-02-19T20:32:20.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:20.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:20.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:20.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:20.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:20.731+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:20.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:20.731+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:21.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:21.280+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:21.281630+0000) 2024-02-19T20:32:21.447 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:21.078+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:21.079676+0000) 2024-02-19T20:32:21.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:21.510+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:21.511658+0000) 2024-02-19T20:32:21.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:21.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:21.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:21.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:21.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:21.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:21.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:21.458+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:21.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:21.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:21.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:21.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:21.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:21.483+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:21.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:21.484+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:22.003 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:21.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:22.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:21.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:22.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:21.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:22.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:21.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:22.272 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:22.082+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:22.083120+0000) 2024-02-19T20:32:22.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:22.514+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:22.515335+0000) 2024-02-19T20:32:22.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:22.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:22.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:22.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:22.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:22.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:22.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:22.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:22.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:22.282+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:22.283740+0000) 2024-02-19T20:32:22.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:22.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:22.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:22.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:22.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:22.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:22.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:22.483+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:22.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:22.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:22.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:22.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:22.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:22.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:22.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:22.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:23.390 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:23.284+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.285863+0000) 2024-02-19T20:32:23.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:23.085+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.086569+0000) 2024-02-19T20:32:23.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:23.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:23.740 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:23.472+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66003 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:23.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:23.468+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.469994+0000) 2024-02-19T20:32:23.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:23.514+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.515753+0000) 2024-02-19T20:32:23.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:23.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:23.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:23.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:23.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:23.401+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:23.742 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:23.469+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.470364+0000) 2024-02-19T20:32:23.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:23.468+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:23.469141+0000) 2024-02-19T20:32:23.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:23.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:23.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:23.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:23.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:23.513+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:23.742 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:23.515+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:24.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:23.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:24.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:23.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:24.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:23.750+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:24.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:23.751+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:24.275 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:24.086+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:24.087220+0000) 2024-02-19T20:32:24.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:24.285+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:24.286306+0000) 2024-02-19T20:32:24.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:24.516+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:24.517912+0000) 2024-02-19T20:32:24.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:24.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:24.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:24.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:24.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:24.406+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:24.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:24.407+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:24.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:24.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:24.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:24.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:24.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:24.751+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:24.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:24.752+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:24.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:24.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:24.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:24.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:24.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:24.565+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:24.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:24.566+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:25.358 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:25.287+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.288379+0000) 2024-02-19T20:32:25.359 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:25.089+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.090808+0000) 2024-02-19T20:32:25.359 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:25.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:25.359 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:25.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:25.359 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:25.369+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:25.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:25.369+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:25.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:25.520+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:25.521559+0000) 2024-02-19T20:32:25.698 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:25.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:25.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:25.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:25.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:25.708+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:25.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:25.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:25.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:25.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:25.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:25.554+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:25.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:25.555+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:26.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:25.709+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:26.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:26.093+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:26.094319+0000) 2024-02-19T20:32:26.530 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:26.522+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:26.523561+0000) 2024-02-19T20:32:26.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:26.289+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:26.290415+0000) 2024-02-19T20:32:26.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:26.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:26.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:26.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:26.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:26.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:26.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:26.404+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:26.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:26.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:26.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:26.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:26.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:26.741+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:26.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:26.742+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:26.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:26.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:26.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:26.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:26.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:26.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:26.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:26.572+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:27.346 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:27.291+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:27.292555+0000) 2024-02-19T20:32:27.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:27.096+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:27.097797+0000) 2024-02-19T20:32:27.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:27.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:27.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:27.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:27.347 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:27.357+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:27.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:27.524+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:27.525661+0000) 2024-02-19T20:32:27.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:27.357+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:27.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:27.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:27.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:27.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:27.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:27.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:27.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:27.572+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:28.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:27.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:28.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:27.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:28.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:27.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:28.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:27.792+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:28.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:28.293+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.294624+0000) 2024-02-19T20:32:28.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:28.100+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.101338+0000) 2024-02-19T20:32:28.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:28.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:28.752 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:28.479+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66043 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:28.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:28.477+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.478285+0000) 2024-02-19T20:32:28.753 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:28.525+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.526069+0000) 2024-02-19T20:32:28.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:28.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:28.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:28.376+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:28.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:28.377+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:28.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:28.476+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.477263+0000) 2024-02-19T20:32:28.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:28.476+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:28.477492+0000) 2024-02-19T20:32:28.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:28.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:28.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:28.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:28.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:28.532+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:28.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:28.533+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:29.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:28.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:29.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:28.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:29.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:28.762+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:29.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:28.763+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:29.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:29.100+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:29.101954+0000) 2024-02-19T20:32:29.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:29.528+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:29.529659+0000) 2024-02-19T20:32:29.541 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:29.294+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:29.295223+0000) 2024-02-19T20:32:29.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:29.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:29.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:29.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:29.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:29.391+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:29.542 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:29.391+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:29.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:29.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:29.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:29.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:29.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:29.717+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:29.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:29.718+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:29.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:29.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:29.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:29.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:29.810 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:29.566+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:29.811 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:29.567+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:30.407 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:30.296+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.297763+0000) 2024-02-19T20:32:30.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:30.103+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.104429+0000) 2024-02-19T20:32:30.688 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:30.532+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:30.533142+0000) 2024-02-19T20:32:30.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:30.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:30.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:30.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:30.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:30.417+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:30.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:30.418+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:30.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:30.523+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:30.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:30.523+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:30.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:30.523+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:30.690 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:30.524+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:31.004 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:30.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:31.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:30.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:31.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:30.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:31.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:30.699+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:31.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:31.107+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:31.108079+0000) 2024-02-19T20:32:31.540 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:31.535+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:31.536679+0000) 2024-02-19T20:32:31.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:31.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:31.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:31.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:31.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:31.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:31.541 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:31.452+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:31.299+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:31.300313+0000) 2024-02-19T20:32:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:31.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:31.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:31.477+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:31.542 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:31.478+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:31.809 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:31.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:31.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:31.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:31.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:31.656+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:31.810 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:31.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:32.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:32.109+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:32.110192+0000) 2024-02-19T20:32:32.407 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:32.301+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:32.302474+0000) 2024-02-19T20:32:32.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:32.433+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:32.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:32.433+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:32.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:32.433+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:32.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:32.433+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:32.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:32.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:32.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:32.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:32.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:32.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:32.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:32.417+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:32.537+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:32.538742+0000) 2024-02-19T20:32:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:32.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:32.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:32.639+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:32.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:32.640+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:33.396 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:33.303+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.304493+0000) 2024-02-19T20:32:33.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:33.111+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.112336+0000) 2024-02-19T20:32:33.780 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:33.487+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66082 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:33.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:33.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:33.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:33.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:33.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:33.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:33.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:33.407+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:33.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:33.483+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.484931+0000) 2024-02-19T20:32:33.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:33.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:33.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:33.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:33.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:33.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:33.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:33.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:33.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:33.484+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.485156+0000) 2024-02-19T20:32:33.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:33.484+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.485515+0000) 2024-02-19T20:32:33.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:33.538+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:33.539174+0000) 2024-02-19T20:32:33.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:33.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:33.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:33.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:33.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:33.632+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:33.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:33.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:34.393 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:34.111+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:34.112898+0000) 2024-02-19T20:32:34.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:34.304+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:34.305034+0000) 2024-02-19T20:32:34.394 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:34.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:34.394 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:34.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:34.394 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:34.403+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:34.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:34.404+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:34.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:34.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:34.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:34.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:34.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:34.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:34.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:34.452+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:34.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:34.540+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:34.541770+0000) 2024-02-19T20:32:34.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:34.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:34.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:34.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:34.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:34.587+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:34.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:34.588+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:35.364 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:35.113+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.114925+0000) 2024-02-19T20:32:35.364 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:35.307+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.308542+0000) 2024-02-19T20:32:35.365 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:35.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:35.365 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:35.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:35.365 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:35.374+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:35.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:35.375+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:35.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:35.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:35.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:35.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:35.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:35.456+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:35.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:35.456+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:35.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:35.544+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:35.545406+0000) 2024-02-19T20:32:35.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:35.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:35.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:35.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:35.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:35.622+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:35.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:35.623+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:36.411 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:36.115+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:36.116971+0000) 2024-02-19T20:32:36.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:36.310+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:36.311976+0000) 2024-02-19T20:32:36.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:36.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:36.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:36.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:36.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:36.422+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:36.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:36.423+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:36.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:36.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:36.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:36.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:36.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:36.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:36.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:36.494+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:36.547+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:36.548980+0000) 2024-02-19T20:32:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:36.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:36.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:36.633+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:36.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:36.633+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:37.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:37.118+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:37.119047+0000) 2024-02-19T20:32:37.381 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:37.313+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:37.314068+0000) 2024-02-19T20:32:37.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:37.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:37.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:37.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:37.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:37.391+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:37.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:37.392+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:37.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:37.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:37.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:37.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:37.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:37.525+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:37.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:37.525+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:37.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:37.550+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:37.551046+0000) 2024-02-19T20:32:37.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:37.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:37.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:37.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:37.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:37.601+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:37.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:37.602+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:38.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:38.121+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.122742+0000) 2024-02-19T20:32:38.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:38.315+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.316138+0000) 2024-02-19T20:32:38.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:38.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:38.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:38.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:38.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:38.406+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:38.780 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:38.495+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66122 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:38.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:38.407+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:38.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:38.492+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.493063+0000) 2024-02-19T20:32:38.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:38.493+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.494343+0000) 2024-02-19T20:32:38.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:38.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:38.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:38.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:38.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:38.498+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:38.782 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:38.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:38.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:38.492+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.493754+0000) 2024-02-19T20:32:38.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:38.550+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:38.551682+0000) 2024-02-19T20:32:38.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:38.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:38.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:38.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:38.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:38.603+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:38.783 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:38.604+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:39.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:39.122+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:39.123388+0000) 2024-02-19T20:32:39.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:39.315+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:39.316709+0000) 2024-02-19T20:32:39.419 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:39.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:39.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:39.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:39.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:39.429+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:39.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:39.430+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:39.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:39.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:39.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:39.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:39.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:39.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:39.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:39.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:39.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:39.554+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:39.555265+0000) 2024-02-19T20:32:39.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:39.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:39.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:39.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:39.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:39.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:39.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:39.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:40.396 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:40.127+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.127067+0000) 2024-02-19T20:32:40.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:40.317+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.318946+0000) 2024-02-19T20:32:40.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:40.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:40.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:40.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:40.397 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:40.407+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:40.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:40.407+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:40.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:40.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:40.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:40.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:40.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:40.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:40.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:40.437+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:40.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:40.556+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:40.557335+0000) 2024-02-19T20:32:40.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:40.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:40.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:40.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:40.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:40.635+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:40.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:40.635+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:41.410 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:41.129+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:41.129052+0000) 2024-02-19T20:32:41.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:41.321+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:41.321197+0000) 2024-02-19T20:32:41.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:41.421+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:41.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:41.421+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:41.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:41.421+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:41.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:41.559+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:41.559499+0000) 2024-02-19T20:32:41.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:41.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:41.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:41.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:41.661 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:41.671+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:41.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:41.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:41.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:41.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:41.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:41.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:41.662 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:41.475+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:41.662 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:41.422+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:42.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:41.672+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:42.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:42.130+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:42.131134+0000) 2024-02-19T20:32:42.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:42.562+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:42.563103+0000) 2024-02-19T20:32:42.592 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:42.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:42.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:42.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:42.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:42.482+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:42.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:42.483+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:42.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:42.322+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:42.323618+0000) 2024-02-19T20:32:42.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:42.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:42.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:42.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:42.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:42.431+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:42.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:42.432+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:43.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:42.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:43.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:42.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:43.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:42.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:43.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:42.644+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:43.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:43.132+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.133200+0000) 2024-02-19T20:32:43.592 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:43.503+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66161 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:43.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:43.499+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.500433+0000) 2024-02-19T20:32:43.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:43.562+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.563547+0000) 2024-02-19T20:32:43.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:43.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:43.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:43.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:43.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:43.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:43.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:43.475+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:43.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:43.499+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.500577+0000) 2024-02-19T20:32:43.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:43.326+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.327247+0000) 2024-02-19T20:32:43.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:43.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:43.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:43.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:43.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:43.470+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:43.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:43.470+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:43.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:43.500+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:43.501677+0000) 2024-02-19T20:32:44.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:43.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:44.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:43.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:44.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:43.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:44.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:43.664+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:44.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:44.132+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:44.133661+0000) 2024-02-19T20:32:44.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:44.564+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:44.565560+0000) 2024-02-19T20:32:44.592 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:44.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:44.592 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:44.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:44.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:44.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:44.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:44.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:44.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:44.326+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:44.327685+0000) 2024-02-19T20:32:44.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:44.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:44.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:44.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:44.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:44.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:44.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:44.465+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:45.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:44.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:45.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:44.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:45.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:44.617+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:45.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:44.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:45.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:45.136+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.137248+0000) 2024-02-19T20:32:45.572 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:45.566+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.567560+0000) 2024-02-19T20:32:45.572 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:45.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:45.572 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:45.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:45.572 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:45.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:45.573 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:45.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:45.573 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:45.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:45.573 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:45.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:45.573 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:45.478+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:45.573 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:45.328+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:45.329957+0000) 2024-02-19T20:32:45.574 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:45.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:45.574 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:45.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:45.574 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:45.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:45.574 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:45.474+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:46.005 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:45.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:46.283 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:46.138+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:46.139229+0000) 2024-02-19T20:32:46.592 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:46.568+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:46.569653+0000) 2024-02-19T20:32:46.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:46.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:46.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:46.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:46.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:46.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:46.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:46.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:46.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:46.331+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:46.332014+0000) 2024-02-19T20:32:46.593 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:46.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:46.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:46.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:46.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:46.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:46.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:46.486+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:47.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:46.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:47.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:46.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:47.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:46.605+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:47.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:46.606+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:47.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:47.140+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:47.141330+0000) 2024-02-19T20:32:47.592 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:47.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:47.592 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:47.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:47.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:47.491+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:47.593 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:47.491+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:47.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:47.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:47.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:47.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:47.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:47.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:47.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:47.557+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:47.594 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:47.570+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:47.571705+0000) 2024-02-19T20:32:47.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:47.333+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:47.334729+0000) 2024-02-19T20:32:47.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:47.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:47.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:47.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:47.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:47.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:47.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:47.508+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:48.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:48.142+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.143302+0000) 2024-02-19T20:32:48.592 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:48.511+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66202 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:48.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:48.507+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.508947+0000) 2024-02-19T20:32:48.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:48.571+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.572167+0000) 2024-02-19T20:32:48.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:48.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:48.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:48.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:48.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:48.575+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:48.593 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:48.576+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:48.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:48.336+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.337316+0000) 2024-02-19T20:32:48.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:48.508+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.509301+0000) 2024-02-19T20:32:48.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:48.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:48.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:48.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:48.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:48.530+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:48.594 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:48.531+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:48.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:48.507+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:48.508319+0000) 2024-02-19T20:32:48.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:48.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:48.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:48.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:48.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:48.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:48.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:48.508+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:49.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:49.142+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:49.143961+0000) 2024-02-19T20:32:49.585 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:49.573+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:49.574225+0000) 2024-02-19T20:32:49.586 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:49.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:49.586 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:49.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:49.586 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:49.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:49.586 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:49.527+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:49.586 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:49.336+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:49.337945+0000) 2024-02-19T20:32:49.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:49.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:49.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:49.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:49.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:49.506+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:49.587 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:49.507+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:50.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:49.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:50.006 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:49.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:50.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:49.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:50.007 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:49.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:50.260 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:50.145+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.146410+0000) 2024-02-19T20:32:50.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:50.339+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.340379+0000) 2024-02-19T20:32:50.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:50.462+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:50.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:50.462+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:50.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:50.462+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:50.531 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:50.463+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:50.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:50.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:50.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:50.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:50.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:50.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:50.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:50.481+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:50.811 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:50.575+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:50.576764+0000) 2024-02-19T20:32:50.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:50.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:50.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:50.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:50.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:50.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:50.812 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:50.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:51.436 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:51.341+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:51.342839+0000) 2024-02-19T20:32:51.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:51.147+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:51.148882+0000) 2024-02-19T20:32:51.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:51.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:51.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:51.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:51.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:51.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:51.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:51.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:51.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:51.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:51.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:51.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:51.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:51.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:51.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:51.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:51.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:51.577+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:51.578824+0000) 2024-02-19T20:32:51.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:51.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:51.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:51.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:51.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:51.590+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:51.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:51.590+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:52.424 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:52.343+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:52.344894+0000) 2024-02-19T20:32:52.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:52.149+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:52.150892+0000) 2024-02-19T20:32:52.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:52.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:52.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:52.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:52.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:52.434+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:52.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:52.435+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:52.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:52.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:52.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:52.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:52.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:52.469+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:52.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:52.470+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:52.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:52.580+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:52.581411+0000) 2024-02-19T20:32:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:52.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:52.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:52.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:52.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:52.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:53.385 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:53.346+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.347176+0000) 2024-02-19T20:32:53.386 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:53.152+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.153081+0000) 2024-02-19T20:32:53.386 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:53.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:53.386 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:53.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:53.386 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:53.396+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:53.780 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:53.520+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66241 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:53.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:53.397+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:53.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:53.516+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.517834+0000) 2024-02-19T20:32:53.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:53.516+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.517801+0000) 2024-02-19T20:32:53.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:53.580+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.581871+0000) 2024-02-19T20:32:53.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:53.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:53.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:53.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:53.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:53.607+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:53.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:53.608+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:53.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:53.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:53.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:53.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:53.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:53.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:53.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:53.456+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:53.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:53.516+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:53.517500+0000) 2024-02-19T20:32:54.372 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:54.346+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:54.347659+0000) 2024-02-19T20:32:54.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:54.152+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:54.153551+0000) 2024-02-19T20:32:54.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:54.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:54.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:54.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:54.373 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:54.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:54.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:54.383+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:54.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:54.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:54.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:54.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:54.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:54.471+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:54.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:54.472+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:54.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:54.582+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:54.583898+0000) 2024-02-19T20:32:54.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:54.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:54.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:54.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:54.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:54.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:54.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:54.621+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:55.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:55.154+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.155958+0000) 2024-02-19T20:32:55.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:55.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:55.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:55.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:55.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:55.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:55.448 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:55.353+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:55.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:55.348+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.349760+0000) 2024-02-19T20:32:55.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:55.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:55.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:55.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:55.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:55.458+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:55.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:55.459+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:55.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:55.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:55.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:55.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:55.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:55.577+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:55.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:55.578+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:55.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:55.584+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:55.585948+0000) 2024-02-19T20:32:56.453 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:56.350+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:56.351848+0000) 2024-02-19T20:32:56.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:56.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:56.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:56.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:56.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:56.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:56.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:56.157+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:56.158146+0000) 2024-02-19T20:32:56.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:56.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:56.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:56.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:56.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:56.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:56.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:56.308+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:56.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:56.465+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:56.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:56.586+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:56.588000+0000) 2024-02-19T20:32:56.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:56.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:56.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:56.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:56.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:56.592+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:56.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:56.592+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:57.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:57.159+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:57.160175+0000) 2024-02-19T20:32:57.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:57.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:57.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:57.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:57.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:57.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:57.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:57.326+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:57.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:57.354+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:57.355456+0000) 2024-02-19T20:32:57.421 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:57.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:57.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:57.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:57.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:57.430+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:57.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:57.431+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:57.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:57.590+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:57.591562+0000) 2024-02-19T20:32:57.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:57.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:57.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:57.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:57.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:57.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:57.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:57.641+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:58.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:58.358+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.359083+0000) 2024-02-19T20:32:58.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:58.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:58.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:58.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:58.467 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:58.161+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.162262+0000) 2024-02-19T20:32:58.467 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:58.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:58.467 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:58.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:58.468 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:58.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:58.468 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:58.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:58.780 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:32:58.528+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66281 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:32:58.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:58.525+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.526912+0000) 2024-02-19T20:32:58.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:58.475+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:58.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:58.476+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:58.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:58.524+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.525785+0000) 2024-02-19T20:32:58.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:58.525+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.526283+0000) 2024-02-19T20:32:58.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:58.591+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:58.592056+0000) 2024-02-19T20:32:58.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:58.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:58.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:58.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:58.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:58.602+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:58.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:58.603+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:59.486 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:59.358+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:59.359647+0000) 2024-02-19T20:32:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:59.161+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:59.162711+0000) 2024-02-19T20:32:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:59.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:59.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:59.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:32:59.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:32:59.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:32:59.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:59.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:59.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:59.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:59.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:59.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:32:59.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:32:59.508+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:32:59.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:59.593+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:32:59.594178+0000) 2024-02-19T20:32:59.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:59.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:59.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:59.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:59.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:59.641+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:32:59.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:32:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:32:59.642+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:00.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:00.361+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.362082+0000) 2024-02-19T20:33:00.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:00.163+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.164733+0000) 2024-02-19T20:33:00.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:00.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:00.475 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:00.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:00.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:00.271+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:00.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:00.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:00.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:00.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:00.485+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:00.486+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:00.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:00.595+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:00.596369+0000) 2024-02-19T20:33:00.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:00.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:00.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:00.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:00.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:00.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:00.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:00.597+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:01.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:01.363+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:01.364687+0000) 2024-02-19T20:33:01.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:01.166+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:01.167264+0000) 2024-02-19T20:33:01.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:01.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:01.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:01.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:01.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:01.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:01.509 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:01.243+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:01.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:01.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:01.765 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:01.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:01.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:01.519+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:01.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:01.520+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:01.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:01.599+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:01.600028+0000) 2024-02-19T20:33:01.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:01.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:01.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:01.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:01.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:01.599+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:01.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:01.600+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:02.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:02.366+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:02.367243+0000) 2024-02-19T20:33:02.521 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:02.168+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:02.169944+0000) 2024-02-19T20:33:02.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:02.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:02.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:02.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:02.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:02.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:02.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:02.248+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:02.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:02.531+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:02.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:02.531+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:02.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:02.531+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:02.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:02.532+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:02.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:02.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:02.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:02.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:02.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:02.586+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:02.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:02.587+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:02.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:02.601+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:02.602769+0000) 2024-02-19T20:33:03.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:03.368+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.369746+0000) 2024-02-19T20:33:03.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:03.171+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.172726+0000) 2024-02-19T20:33:03.519 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:03.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:03.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:03.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:03.520 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:03.253+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:03.780 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:03.536+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66320 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:03.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:03.533+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.534570+0000) 2024-02-19T20:33:03.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:03.533+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.534433+0000) 2024-02-19T20:33:03.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:03.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:03.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:03.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:03.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:03.580+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:03.781 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:03.581+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:03.782 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:03.602+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.603228+0000) 2024-02-19T20:33:03.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:03.532+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:03.533314+0000) 2024-02-19T20:33:03.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:03.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:03.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:03.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:03.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:03.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:03.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:03.547+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:04.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:04.369+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:04.370292+0000) 2024-02-19T20:33:04.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:04.172+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:04.173359+0000) 2024-02-19T20:33:04.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:04.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:04.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:04.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:04.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:04.289+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:04.532 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:04.289+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:04.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:04.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:04.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:04.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:04.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:04.574+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:04.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:04.575+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:04.813 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:04.604+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:04.605193+0000) 2024-02-19T20:33:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:04.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:04.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:04.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:04.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:04.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:05.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:05.371+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.372321+0000) 2024-02-19T20:33:05.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:05.174+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.175357+0000) 2024-02-19T20:33:05.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:05.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:05.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:05.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:05.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:05.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:05.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:05.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:05.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:05.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:05.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:05.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:05.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:05.573+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:05.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:05.574+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:05.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:05.606+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:05.607209+0000) 2024-02-19T20:33:05.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:05.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:05.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:05.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:05.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:05.608+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:05.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:05.609+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:06.364 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:06.176+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:06.177347+0000) 2024-02-19T20:33:06.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:06.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:06.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:06.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:06.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:06.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:06.365 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:06.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:06.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:06.608+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:06.609668+0000) 2024-02-19T20:33:06.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:06.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:06.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:06.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:06.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:06.640+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:06.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:06.374+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:06.375920+0000) 2024-02-19T20:33:06.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:06.602+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:06.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:06.602+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:06.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:06.602+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:06.631 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:06.603+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:07.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:06.641+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:07.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:07.179+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:07.180788+0000) 2024-02-19T20:33:07.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:07.377+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:07.378354+0000) 2024-02-19T20:33:07.612 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:07.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:07.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:07.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:07.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:07.597+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:07.613 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:07.598+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:07.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:07.610+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:07.611745+0000) 2024-02-19T20:33:07.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:07.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:07.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:07.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:07.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:07.621+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:07.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:07.622+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:07.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:07.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:07.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:07.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:07.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:07.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:07.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:07.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:08.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:08.183+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.184305+0000) 2024-02-19T20:33:08.595 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:08.544+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66360 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:08.595 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:08.541+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.542377+0000) 2024-02-19T20:33:08.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:08.379+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.380464+0000) 2024-02-19T20:33:08.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:08.541+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.542707+0000) 2024-02-19T20:33:08.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:08.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:08.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:08.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:08.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:08.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:08.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:08.346+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:08.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:08.541+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.542322+0000) 2024-02-19T20:33:09.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:08.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:09.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:08.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:09.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:08.648+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:09.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:08.648+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:09.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:08.611+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:08.612218+0000) 2024-02-19T20:33:09.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:08.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:09.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:08.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:09.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:08.657+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:09.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:08.657+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:09.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:09.183+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:09.184788+0000) 2024-02-19T20:33:09.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:09.380+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:09.381055+0000) 2024-02-19T20:33:09.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:09.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:09.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:09.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:09.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:09.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:09.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:09.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:10.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:09.614+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:09.615665+0000) 2024-02-19T20:33:10.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:09.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:10.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:09.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:10.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:09.681+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:10.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:09.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:10.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:09.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:10.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:09.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:10.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:09.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:10.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:09.681+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:10.261 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:10.185+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.186880+0000) 2024-02-19T20:33:10.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:10.382+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.383480+0000) 2024-02-19T20:33:10.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:10.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:10.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:10.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:10.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:10.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:10.531 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:10.364+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:10.813 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:10.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:10.829 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:10.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:10.829 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:10.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:10.829 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:10.673+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:10.829 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:10.616+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:10.617592+0000) 2024-02-19T20:33:10.830 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:10.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:10.830 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:10.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:10.830 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:10.701+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:10.830 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:10.702+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:11.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:11.188+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:11.189063+0000) 2024-02-19T20:33:11.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:11.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:11.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:11.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:11.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:11.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:11.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:11.324+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:11.671 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:11.385+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:11.386932+0000) 2024-02-19T20:33:11.671 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:11.618+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:11.619583+0000) 2024-02-19T20:33:11.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:11.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:11.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:11.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:11.672 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:11.682+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:12.008 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:11.682+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:12.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:11.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:12.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:11.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:12.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:11.683+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:12.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:11.684+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:12.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:12.190+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:12.191167+0000) 2024-02-19T20:33:12.595 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:12.388+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:12.389146+0000) 2024-02-19T20:33:12.595 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:12.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:12.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:12.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:12.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:12.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:12.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:12.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:13.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:12.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:13.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:12.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:13.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:12.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:13.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:12.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:13.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:12.621+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:12.622039+0000) 2024-02-19T20:33:13.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:12.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:13.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:12.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:13.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:12.696+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:13.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:12.696+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:13.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:13.192+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.193214+0000) 2024-02-19T20:33:13.535 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:13.390+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.391215+0000) 2024-02-19T20:33:13.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:13.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:13.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:13.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:13.535 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:13.293+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:13.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:13.293+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:13.813 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:13.552+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66399 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:13.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:13.548+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.549515+0000) 2024-02-19T20:33:13.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:13.549+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.550612+0000) 2024-02-19T20:33:13.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:13.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:13.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:13.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:13.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:13.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:13.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:13.627+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:13.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:13.549+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.550324+0000) 2024-02-19T20:33:13.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:13.621+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:13.622504+0000) 2024-02-19T20:33:13.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:13.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:13.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:13.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:13.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:13.698+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:13.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:13.699+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:14.379 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:14.192+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:14.193738+0000) 2024-02-19T20:33:14.380 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:14.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:14.380 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:14.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:14.380 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:14.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:14.380 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:14.270+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:14.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:14.624+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:14.625904+0000) 2024-02-19T20:33:14.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:14.390+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:14.391708+0000) 2024-02-19T20:33:14.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:14.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:14.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:14.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:15.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:14.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:15.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:14.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:15.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:14.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:15.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:14.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:15.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:14.744+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:15.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:14.745+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:15.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:15.195+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.196172+0000) 2024-02-19T20:33:15.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:15.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:15.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:15.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:15.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:15.265+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:15.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:15.266+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:15.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:15.392+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.393754+0000) 2024-02-19T20:33:15.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:15.626+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:15.627870+0000) 2024-02-19T20:33:16.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:15.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:16.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:15.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:16.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:15.778+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:16.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:15.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:16.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:15.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:16.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:15.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:16.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:15.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:16.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:15.707+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:16.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:16.197+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:16.198151+0000) 2024-02-19T20:33:16.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:16.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:16.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:16.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:16.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:16.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:16.281 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:16.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:16.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:16.629+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:16.630126+0000) 2024-02-19T20:33:16.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:16.394+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:16.395863+0000) 2024-02-19T20:33:16.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:16.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:16.731 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:16.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:16.731 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:16.741+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:17.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:16.741+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:17.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:16.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:17.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:16.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:17.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:16.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:17.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:16.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:17.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:17.199+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:17.200293+0000) 2024-02-19T20:33:17.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:17.396+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:17.397943+0000) 2024-02-19T20:33:17.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:17.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:17.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:17.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:17.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:17.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:17.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:17.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:17.814 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:17.631+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:17.632742+0000) 2024-02-19T20:33:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:17.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:17.814 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:17.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:17.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:17.696+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:17.815 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:17.697+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:18.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:17.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:18.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:17.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:18.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:17.858+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:18.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:17.858+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:18.388 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:18.201+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.202445+0000) 2024-02-19T20:33:18.388 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:18.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:18.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:18.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:18.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:18.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:18.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:18.313+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:18.648 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:18.560+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66439 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:18.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:18.557+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.558133+0000) 2024-02-19T20:33:18.649 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:18.557+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.558893+0000) 2024-02-19T20:33:18.651 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:18.632+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.633214+0000) 2024-02-19T20:33:18.651 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:18.399+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.400080+0000) 2024-02-19T20:33:18.652 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:18.557+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:18.558504+0000) 2024-02-19T20:33:18.652 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:18.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:18.652 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:18.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:18.652 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:18.658+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:19.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:18.659+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:19.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:18.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:19.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:18.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:19.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:18.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:19.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:18.883+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:19.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:19.201+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:19.202948+0000) 2024-02-19T20:33:19.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:19.399+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:19.400724+0000) 2024-02-19T20:33:19.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:19.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:19.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:19.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:19.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:19.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:19.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:19.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:19.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:19.634+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:19.635548+0000) 2024-02-19T20:33:19.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:19.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:19.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:19.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:19.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:19.869+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:19.859 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:19.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:19.859 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:19.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:19.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:19.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:19.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:19.682+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:20.114 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:19.870+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:20.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:20.204+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.205060+0000) 2024-02-19T20:33:20.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:20.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:20.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:20.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:20.393 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:20.296+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:20.393 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:20.296+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:20.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:20.637+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.638062+0000) 2024-02-19T20:33:20.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:20.403+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:20.404169+0000) 2024-02-19T20:33:21.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:20.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:21.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:20.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:21.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:20.669+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:21.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:20.670+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:21.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:20.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:21.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:20.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:21.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:20.872+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:21.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:20.873+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:21.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:21.206+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:21.207068+0000) 2024-02-19T20:33:21.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:21.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:21.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:21.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:21.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:21.247+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:21.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:21.247+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:21.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:21.639+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:21.640269+0000) 2024-02-19T20:33:21.670 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:21.405+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:21.406655+0000) 2024-02-19T20:33:22.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:21.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:22.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:21.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:22.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:21.681+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:22.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:21.682+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:22.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:21.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:22.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:21.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:22.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:21.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:22.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:21.830+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:22.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:22.208+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:22.209548+0000) 2024-02-19T20:33:22.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:22.246+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:22.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:22.246+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:22.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:22.246+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:22.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:22.247+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:22.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:22.641+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:22.642303+0000) 2024-02-19T20:33:22.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:22.409+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:22.410170+0000) 2024-02-19T20:33:22.693 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:22.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:22.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:22.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:22.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:22.704+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:23.009 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:22.704+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:23.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:22.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:23.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:22.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:23.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:22.876+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:23.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:22.877+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:23.269 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:23.210+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.211649+0000) 2024-02-19T20:33:23.530 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:23.412+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.413616+0000) 2024-02-19T20:33:23.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:23.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:23.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:23.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:23.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:23.282+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:23.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:23.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:23.815 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:23.568+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66478 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:23.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:23.564+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.565595+0000) 2024-02-19T20:33:23.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:23.641+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.642830+0000) 2024-02-19T20:33:23.816 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:23.565+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.566689+0000) 2024-02-19T20:33:23.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:23.564+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:23.565249+0000) 2024-02-19T20:33:23.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:23.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:23.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:23.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:23.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:23.702+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:23.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:23.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:24.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:23.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:24.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:23.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:24.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:23.856+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:24.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:23.857+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:24.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:24.211+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:24.212120+0000) 2024-02-19T20:33:24.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:24.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:24.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:24.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:24.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:24.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:24.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:24.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:24.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:24.644+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:24.645366+0000) 2024-02-19T20:33:24.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:24.413+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:24.414396+0000) 2024-02-19T20:33:24.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:24.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:24.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:24.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:25.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:24.679+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:25.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:24.680+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:25.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:24.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:25.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:24.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:25.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:24.840+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:25.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:24.841+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:25.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:25.213+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.214327+0000) 2024-02-19T20:33:25.596 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:25.415+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.417001+0000) 2024-02-19T20:33:25.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:25.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:25.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:25.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:25.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:25.337+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:25.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:25.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:26.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:25.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:26.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:25.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:26.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:25.725+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:26.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:25.725+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:26.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:25.646+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:25.647371+0000) 2024-02-19T20:33:26.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:25.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:26.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:25.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:26.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:25.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:26.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:25.828+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:26.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:26.216+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:26.217935+0000) 2024-02-19T20:33:26.597 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:26.419+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:26.419410+0000) 2024-02-19T20:33:26.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:26.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:26.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:26.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:26.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:26.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:26.597 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:26.348+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:27.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:26.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:27.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:26.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:27.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:26.769+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:27.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:26.769+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:27.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:26.651+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:26.651099+0000) 2024-02-19T20:33:27.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:26.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:27.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:26.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:27.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:26.779+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:27.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:26.779+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:27.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:27.220+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:27.221662+0000) 2024-02-19T20:33:27.641 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:27.422+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:27.422156+0000) 2024-02-19T20:33:27.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:27.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:27.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:27.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:27.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:27.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:27.642 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:27.385+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:28.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:27.653+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:27.653273+0000) 2024-02-19T20:33:28.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:27.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:28.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:27.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:28.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:27.794+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:28.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:27.795+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:28.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:27.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:28.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:27.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:28.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:27.759+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:28.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:27.760+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:28.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:28.225+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.225344+0000) 2024-02-19T20:33:28.682 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:28.572+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.573748+0000) 2024-02-19T20:33:28.682 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:28.652+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.653766+0000) 2024-02-19T20:33:28.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:28.423+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.424229+0000) 2024-02-19T20:33:28.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:28.572+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.573672+0000) 2024-02-19T20:33:28.683 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:28.575+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66518 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:28.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:28.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:28.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:28.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:28.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:28.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:28.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:28.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:28.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:28.572+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:28.573814+0000) 2024-02-19T20:33:29.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:28.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:29.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:28.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:29.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:28.793+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:29.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:28.794+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:29.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:28.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:29.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:28.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:29.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:28.743+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:29.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:28.743+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:29.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:29.225+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:29.226030+0000) 2024-02-19T20:33:29.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:29.654+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:29.655824+0000) 2024-02-19T20:33:29.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:29.423+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:29.424752+0000) 2024-02-19T20:33:29.692 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:29.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:29.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:29.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:29.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:29.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:29.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:29.451+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:29.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:29.452+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:30.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:29.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:30.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:29.703+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:30.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:29.703+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:30.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:29.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:30.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:29.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:30.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:29.833+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:30.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:29.833+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:30.262 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:30.228+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.229643+0000) 2024-02-19T20:33:30.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:30.426+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.427163+0000) 2024-02-19T20:33:30.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:30.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:30.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:30.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:30.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:30.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:30.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:30.403+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:30.798 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:30.657+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:30.658360+0000) 2024-02-19T20:33:30.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:30.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:30.799 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:30.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:30.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:30.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:30.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:30.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:30.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:30.662+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:30.799 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:30.662+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:31.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:30.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:31.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:30.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:31.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:31.230+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:31.231711+0000) 2024-02-19T20:33:31.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:31.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:31.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:31.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:31.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:31.354+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:31.418 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:31.355+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:31.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:31.659+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:31.660403+0000) 2024-02-19T20:33:31.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:31.428+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:31.429165+0000) 2024-02-19T20:33:31.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:31.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:31.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:31.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:31.783 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:31.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:31.784 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:31.650+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:32.115 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:31.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:32.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:31.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:32.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:31.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:32.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:31.859+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:32.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:32.232+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:32.233792+0000) 2024-02-19T20:33:32.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:32.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:32.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:32.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:32.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:32.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:32.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:32.320+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:32.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:32.661+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:32.662351+0000) 2024-02-19T20:33:32.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:32.430+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:32.431611+0000) 2024-02-19T20:33:32.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:32.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:32.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:32.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:32.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:32.638+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:32.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:32.639+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:33.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:32.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:33.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:32.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:33.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:32.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:33.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:32.896+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:33.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:33.236+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.237508+0000) 2024-02-19T20:33:33.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:33.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:33.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:33.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:33.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:33.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:33.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:33.308+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:33.692 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:33.583+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66557 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:33.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:33.579+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.580918+0000) 2024-02-19T20:33:33.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:33.580+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.581359+0000) 2024-02-19T20:33:33.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:33.662+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.663152+0000) 2024-02-19T20:33:33.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:33.432+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.433883+0000) 2024-02-19T20:33:33.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:33.581+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:33.582209+0000) 2024-02-19T20:33:33.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:33.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:33.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:33.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:33.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:33.680+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:33.694 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:33.681+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:34.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:33.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:34.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:33.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:34.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:33.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:34.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:33.857+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:34.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:34.237+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:34.238032+0000) 2024-02-19T20:33:34.597 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:34.433+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:34.434653+0000) 2024-02-19T20:33:34.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:34.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:34.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:34.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:34.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:34.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:34.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:34.317+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:34.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:34.664+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:34.665705+0000) 2024-02-19T20:33:34.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:34.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:34.884 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:34.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:34.884 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:34.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:34.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:34.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:34.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:34.705+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:34.885 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:34.705+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:35.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:34.895+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:35.179 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:34.895+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:35.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:35.435+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.436650+0000) 2024-02-19T20:33:35.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:35.239+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.240218+0000) 2024-02-19T20:33:35.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:35.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:35.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:35.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:35.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:35.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:35.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:35.346+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:35.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:35.667+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:35.668160+0000) 2024-02-19T20:33:35.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:35.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:35.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:35.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:35.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:35.675+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:35.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:35.676+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:36.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:35.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:36.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:35.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:36.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:35.883+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:36.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:35.885+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:36.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:36.241+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:36.242350+0000) 2024-02-19T20:33:36.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:36.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:36.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:36.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:36.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:36.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:36.429 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:36.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:36.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:36.669+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:36.670555+0000) 2024-02-19T20:33:36.689 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:36.439+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:36.440254+0000) 2024-02-19T20:33:37.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:36.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:37.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:36.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:37.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:36.868+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:37.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:36.869+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:37.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:36.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:37.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:36.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:37.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:36.700+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:37.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:36.701+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:37.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:37.245+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:37.246028+0000) 2024-02-19T20:33:37.598 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:37.442+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:37.443423+0000) 2024-02-19T20:33:37.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:37.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:37.598 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:37.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:37.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:37.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:37.599 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:37.356+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:37.889 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:37.671+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:37.672551+0000) 2024-02-19T20:33:37.890 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:37.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:37.890 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:37.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:37.890 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:37.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:37.890 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:37.686+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:38.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:37.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:38.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:37.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:38.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:37.900+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:38.181 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:37.901+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:38.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:38.247+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.248216+0000) 2024-02-19T20:33:38.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:38.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:38.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:38.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:38.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:38.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:38.435 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:38.378+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:38.686 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:38.591+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66597 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:38.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:38.588+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.589341+0000) 2024-02-19T20:33:38.687 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:38.588+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.589515+0000) 2024-02-19T20:33:38.687 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:38.672+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.673144+0000) 2024-02-19T20:33:38.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:38.446+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.447063+0000) 2024-02-19T20:33:38.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:38.587+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:38.588935+0000) 2024-02-19T20:33:38.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:38.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:38.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:38.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:38.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:38.697+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:39.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:38.699+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:39.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:38.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:39.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:38.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:39.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:38.941+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:39.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:38.942+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:39.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:39.248+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:39.249095+0000) 2024-02-19T20:33:39.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:39.674+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:39.675302+0000) 2024-02-19T20:33:39.675 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:39.446+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:39.447857+0000) 2024-02-19T20:33:39.675 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:39.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:39.675 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:39.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:39.675 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:39.685+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:39.676 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:39.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:39.676 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:39.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:39.676 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:39.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:39.676 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:39.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:40.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:39.686+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:40.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:39.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:40.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:39.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:40.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:39.902+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:40.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:39.903+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:40.262 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:40.251+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:40.252815+0000) 2024-02-19T20:33:40.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:40.448+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:40.449922+0000) 2024-02-19T20:33:40.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:40.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:40.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:40.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:40.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:40.425+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:40.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:40.425+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:40.816 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:40.676+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:40.677332+0000) 2024-02-19T20:33:40.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:40.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:40.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:40.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:40.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:40.670+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:40.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:40.671+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:41.116 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:40.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:41.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:40.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:41.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:40.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:41.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:40.858+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:41.439 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:41.255+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.256364+0000) 2024-02-19T20:33:41.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:41.678+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.679526+0000) 2024-02-19T20:33:41.701 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:41.451+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:41.452036+0000) 2024-02-19T20:33:41.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:41.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:41.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:41.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:41.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:41.457+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:41.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:41.458+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:42.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:41.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:42.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:41.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:42.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:41.843+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:42.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:41.844+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:42.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:41.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:42.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:41.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:42.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:41.712+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:42.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:41.713+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:42.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:42.257+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:42.258868+0000) 2024-02-19T20:33:42.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:42.680+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:42.681564+0000) 2024-02-19T20:33:42.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:42.454+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:42.455697+0000) 2024-02-19T20:33:42.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:42.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:42.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:42.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:42.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:42.687+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:42.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:42.687+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:42.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:42.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:42.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:42.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:42.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:42.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:42.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:42.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:43.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:42.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:43.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:42.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:43.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:42.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:43.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:42.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:43.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:43.261+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.262595+0000) 2024-02-19T20:33:43.707 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:43.599+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66636 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:43.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:43.456+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.457783+0000) 2024-02-19T20:33:43.708 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:43.595+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.596532+0000) 2024-02-19T20:33:43.708 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:43.596+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.597445+0000) 2024-02-19T20:33:43.708 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:43.681+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.682173+0000) 2024-02-19T20:33:43.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:43.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:43.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:43.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:43.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:43.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:43.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:43.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:43.709 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:43.597+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:43.598588+0000) 2024-02-19T20:33:44.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:43.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:44.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:43.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:44.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:43.815+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:44.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:43.816+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:44.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:43.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:44.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:43.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:44.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:43.732+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:44.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:43.733+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:44.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:44.262+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:44.263557+0000) 2024-02-19T20:33:44.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:44.683+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:44.684953+0000) 2024-02-19T20:33:44.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:44.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:44.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:44.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:44.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:44.471+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:44.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:44.471+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:44.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:44.457+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:44.458410+0000) 2024-02-19T20:33:44.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:44.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:44.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:44.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:44.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:44.695+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:44.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:44.696+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:45.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:44.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:45.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:44.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:45.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:44.853+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:45.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:44.854+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:45.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:45.264+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:45.265651+0000) 2024-02-19T20:33:45.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:45.685+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:45.686975+0000) 2024-02-19T20:33:45.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:45.460+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:45.461806+0000) 2024-02-19T20:33:45.727 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:45.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:45.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:45.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:45.728 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:45.738+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:45.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:45.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:45.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:45.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:45.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:45.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:45.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:45.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:46.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:45.739+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:46.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:45.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:46.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:45.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:46.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:45.822+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:46.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:45.823+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:46.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:46.266+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.267809+0000) 2024-02-19T20:33:46.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:46.687+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.689008+0000) 2024-02-19T20:33:46.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:46.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:46.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:46.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:46.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:46.481+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:46.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:46.482+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:46.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:46.463+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:46.464714+0000) 2024-02-19T20:33:46.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:46.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:46.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:46.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:46.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:46.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:46.782 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:46.695+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:47.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:46.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:47.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:46.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:47.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:46.821+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:47.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:46.822+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:47.456 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:47.268+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:47.269883+0000) 2024-02-19T20:33:47.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:47.690+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:47.691440+0000) 2024-02-19T20:33:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:47.467+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:47.468271+0000) 2024-02-19T20:33:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:47.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:47.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:47.667+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:47.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:47.668+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:47.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:47.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:47.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:47.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:47.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:47.478+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:47.781 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:47.479+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:48.117 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:47.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:48.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:47.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:48.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:47.809+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:48.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:47.810+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:48.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:48.271+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.272380+0000) 2024-02-19T20:33:48.680 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:48.607+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66676 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:48.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:48.604+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.605105+0000) 2024-02-19T20:33:48.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:48.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:48.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:48.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:48.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:48.439+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:48.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:48.440+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:48.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:48.604+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.605370+0000) 2024-02-19T20:33:48.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:48.470+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.471890+0000) 2024-02-19T20:33:48.681 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:48.604+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.605202+0000) 2024-02-19T20:33:48.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:48.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:48.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:48.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:48.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:48.634+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:48.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:48.635+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:49.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:48.691+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:48.692160+0000) 2024-02-19T20:33:49.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:48.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:49.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:48.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:49.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:48.782+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:49.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:48.783+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:49.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:49.272+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:49.273231+0000) 2024-02-19T20:33:49.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:49.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:49.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:49.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:49.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:49.423+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:49.683 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:49.423+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:49.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:49.471+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:49.472444+0000) 2024-02-19T20:33:49.683 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:49.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:49.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:49.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:49.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:49.630+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:49.684 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:49.631+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:50.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:49.694+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:49.695128+0000) 2024-02-19T20:33:50.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:49.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:50.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:49.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:50.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:49.826+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:50.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:49.827+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:50.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:50.473+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:50.474459+0000) 2024-02-19T20:33:50.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:50.274+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:50.275185+0000) 2024-02-19T20:33:50.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:50.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:50.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:50.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:50.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:50.436+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:50.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:50.436+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:50.817 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:50.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:50.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:50.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:50.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:50.613+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:50.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:50.613+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:50.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:50.696+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:50.697760+0000) 2024-02-19T20:33:50.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:50.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:50.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:50.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:50.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:50.795+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:50.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:50.796+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:51.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:51.276+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.277349+0000) 2024-02-19T20:33:51.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:51.476+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.477418+0000) 2024-02-19T20:33:51.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:51.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:51.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:51.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:51.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:51.649+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:51.688 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:51.650+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:51.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:51.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:51.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:51.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:51.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:51.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:51.689 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:51.447+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:52.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:51.699+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:51.700195+0000) 2024-02-19T20:33:52.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:51.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:52.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:51.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:52.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:51.825+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:52.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:51.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:52.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:52.478+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:52.479682+0000) 2024-02-19T20:33:52.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:52.278+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:52.279338+0000) 2024-02-19T20:33:52.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:52.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:52.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:52.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:52.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:52.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:52.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:52.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:52.817 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:52.701+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:52.702761+0000) 2024-02-19T20:33:52.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:52.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:52.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:52.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:52.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:52.628+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:52.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:52.629+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:53.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:52.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:53.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:52.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:53.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:52.830+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:53.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:52.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:53.447 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:53.280+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.281608+0000) 2024-02-19T20:33:53.447 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:53.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:53.724 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:53.615+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66715 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:53.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:53.611+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.612702+0000) 2024-02-19T20:33:53.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:53.702+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.703406+0000) 2024-02-19T20:33:53.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:53.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:53.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:53.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:53.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:53.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:53.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:53.612+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.613351+0000) 2024-02-19T20:33:53.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:53.482+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.483352+0000) 2024-02-19T20:33:53.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:53.612+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:53.613421+0000) 2024-02-19T20:33:53.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:53.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:53.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:53.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:53.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:53.618+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:53.726 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:53.619+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:54.012 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:53.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:54.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:53.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:54.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:53.786+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:54.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:53.787+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:54.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:54.482+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:54.483905+0000) 2024-02-19T20:33:54.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:54.281+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:54.282179+0000) 2024-02-19T20:33:54.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:54.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:54.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:54.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:54.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:54.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:54.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:54.446+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:54.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:54.704+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:54.705522+0000) 2024-02-19T20:33:54.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:54.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:54.818 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:54.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:54.819 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:54.576+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:54.819 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:54.577+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:55.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:54.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:55.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:54.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:55.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:54.831+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:55.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:54.831+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:55.391 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:55.283+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:55.284341+0000) 2024-02-19T20:33:55.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:55.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:55.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:55.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:55.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:55.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:55.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:55.403+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:55.695 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:55.485+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:55.486376+0000) 2024-02-19T20:33:55.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:55.600+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:55.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:55.600+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:55.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:55.600+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:55.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:55.601+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:56.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:55.706+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:55.707756+0000) 2024-02-19T20:33:56.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:55.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:56.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:55.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:56.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:55.850+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:56.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:55.851+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:56.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:56.487+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.488432+0000) 2024-02-19T20:33:56.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:56.285+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.286395+0000) 2024-02-19T20:33:56.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:56.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:56.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:56.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:56.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:56.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:56.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:56.414+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:56.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:56.709+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:56.710198+0000) 2024-02-19T20:33:56.838 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:56.581+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:56.838 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:56.581+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:56.838 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:56.581+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:56.838 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:56.582+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:57.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:56.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:57.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:56.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:57.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:56.870+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:57.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:56.871+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:57.478 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:57.287+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:57.288725+0000) 2024-02-19T20:33:57.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:57.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:57.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:57.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:57.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:57.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:57.479 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:57.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:57.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:57.712+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:57.713629+0000) 2024-02-19T20:33:57.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:57.489+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:57.490907+0000) 2024-02-19T20:33:57.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:57.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:57.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:57.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:57.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:57.558+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:57.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:57.558+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:58.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:57.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:58.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:57.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:58.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:57.829+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:58.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:57.830+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:58.369 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:58.289+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.290821+0000) 2024-02-19T20:33:58.370 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:58.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:58.622 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:33:58.624+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66755 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:33:58.622 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:58.619+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.620735+0000) 2024-02-19T20:33:58.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:58.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:58.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:58.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:58.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:58.381+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:58.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:58.619+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.620703+0000) 2024-02-19T20:33:58.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:58.493+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.494429+0000) 2024-02-19T20:33:58.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:58.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:58.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:58.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:58.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:58.590+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:58.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:58.591+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:33:58.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:58.620+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.621340+0000) 2024-02-19T20:33:59.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:58.713+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:58.714550+0000) 2024-02-19T20:33:59.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:58.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:59.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:58.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:59.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:58.799+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:59.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:58.800+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:59.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:59.494+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:59.495349+0000) 2024-02-19T20:33:59.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:59.290+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:59.291579+0000) 2024-02-19T20:33:59.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:59.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:59.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:59.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:59.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:59.356+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:33:59.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:33:59.356+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:59.818 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:59.715+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:33:59.716627+0000) 2024-02-19T20:33:59.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:59.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:59.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:59.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:59.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:59.791+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:33:59.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:33:59.792+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:33:59.819 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:59.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:59.819 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:59.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:59.820 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:59.589+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:33:59.820 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:33:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:33:59.590+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:00.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:00.293+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:00.294923+0000) 2024-02-19T20:34:00.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:00.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:00.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:00.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:00.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:00.309+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:00.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:00.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:00.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:00.718+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:00.719055+0000) 2024-02-19T20:34:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:00.497+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:00.498065+0000) 2024-02-19T20:34:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:00.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:00.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:00.572+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:00.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:00.573+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:01.118 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:00.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:01.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:00.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:01.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:00.824+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:01.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:00.825+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:01.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:01.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:01.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:01.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:01.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:01.270+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:01.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:01.270+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:01.488 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:01.296+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.297115+0000) 2024-02-19T20:34:01.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:01.720+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.721129+0000) 2024-02-19T20:34:01.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:01.499+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:01.500225+0000) 2024-02-19T20:34:01.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:01.606+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:01.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:01.606+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:01.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:01.606+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:01.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:01.607+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:02.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:01.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:02.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:01.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:02.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:01.857+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:02.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:01.858+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:02.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:02.298+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:02.299307+0000) 2024-02-19T20:34:02.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:02.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:02.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:02.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:02.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:02.308+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:02.490 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:02.309+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:02.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:02.722+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:02.723090+0000) 2024-02-19T20:34:02.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:02.501+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:02.502304+0000) 2024-02-19T20:34:02.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:02.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:02.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:02.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:02.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:02.642+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:02.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:02.643+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:03.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:02.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:03.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:02.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:03.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:02.828+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:03.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:02.829+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:03.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:03.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:03.261+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:03.262+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:03.492 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:03.301+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.302799+0000) 2024-02-19T20:34:03.779 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:03.632+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66794 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:03.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:03.629+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.630807+0000) 2024-02-19T20:34:03.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:03.628+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.629754+0000) 2024-02-19T20:34:03.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:03.722+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.723679+0000) 2024-02-19T20:34:03.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:03.503+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.504459+0000) 2024-02-19T20:34:03.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:03.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:03.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:03.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:03.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:03.627+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:03.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:03.628+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:03.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:03.630+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:03.631630+0000) 2024-02-19T20:34:04.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:03.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:04.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:03.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:04.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:03.819+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:04.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:03.820+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:04.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:04.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:04.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:04.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:04.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:04.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:04.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:04.249+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:04.493 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:04.302+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:04.303433+0000) 2024-02-19T20:34:04.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:04.726+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:04.727168+0000) 2024-02-19T20:34:04.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:04.504+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:04.505340+0000) 2024-02-19T20:34:04.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:04.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:04.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:04.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:04.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:04.672+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:04.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:04.673+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:05.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:04.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:05.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:04.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:05.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:04.813+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:05.119 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:04.813+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:05.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:05.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:05.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:05.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:05.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:05.256+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:05.495 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:05.256+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:05.496 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:05.306+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:05.307029+0000) 2024-02-19T20:34:05.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:05.728+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:05.729308+0000) 2024-02-19T20:34:05.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:05.506+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:05.507556+0000) 2024-02-19T20:34:05.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:05.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:05.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:05.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:05.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:05.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:05.755 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:05.695+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:06.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:05.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:06.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:05.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:06.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:05.766+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:06.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:05.766+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:06.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:06.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:06.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:06.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:06.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:06.257+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:06.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:06.258+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:06.551 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:06.509+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.510245+0000) 2024-02-19T20:34:06.552 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:06.309+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.310668+0000) 2024-02-19T20:34:06.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:06.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:06.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:06.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:06.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:06.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:06.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:06.727+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:06.820 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:06.730+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:06.731394+0000) 2024-02-19T20:34:06.820 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:06.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:06.820 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:06.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:06.820 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:06.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:06.821 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:06.731+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:07.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:07.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:07.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:07.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:07.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:07.252+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:07.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:07.252+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:07.501 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:07.312+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:07.313379+0000) 2024-02-19T20:34:07.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:07.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:07.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:07.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:07.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:07.728+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:07.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:07.729+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:07.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:07.733+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:07.734055+0000) 2024-02-19T20:34:07.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:07.511+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:07.512746+0000) 2024-02-19T20:34:07.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:07.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:07.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:07.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:07.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:07.730+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:07.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:07.731+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:08.502 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:08.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:08.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:08.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:08.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:08.215+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:08.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:08.215+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:08.503 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:08.314+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.315439+0000) 2024-02-19T20:34:08.779 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:08.641+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66834 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:08.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:08.635+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.636968+0000) 2024-02-19T20:34:08.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:08.636+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.637997+0000) 2024-02-19T20:34:08.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:08.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:08.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:08.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:08.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:08.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:08.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:08.701+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:08.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:08.733+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.734631+0000) 2024-02-19T20:34:08.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:08.513+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.514989+0000) 2024-02-19T20:34:08.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:08.636+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:08.637542+0000) 2024-02-19T20:34:08.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:08.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:08.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:08.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:08.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:08.723+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:08.781 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:08.724+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:09.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:09.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:09.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:09.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:09.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:09.237+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:09.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:09.237+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:09.487 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:09.315+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:09.316262+0000) 2024-02-19T20:34:09.736 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:09.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:09.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:09.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:09.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:09.667+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:09.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:09.667+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:09.737 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:09.737+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:09.738137+0000) 2024-02-19T20:34:09.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:09.514+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:09.515925+0000) 2024-02-19T20:34:09.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:09.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:09.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:09.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:09.737 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:09.747+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:09.738 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:09.748+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:10.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:10.233+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:10.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:10.233+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:10.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:10.233+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:10.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:10.233+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:10.602 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:10.317+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:10.318185+0000) 2024-02-19T20:34:10.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:10.517+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:10.518361+0000) 2024-02-19T20:34:11.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:10.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:11.014 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:10.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:11.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:10.715+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:11.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:10.716+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:11.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:10.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:11.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:10.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:11.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:10.700+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:11.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:10.701+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:11.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:10.739+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:10.740161+0000) 2024-02-19T20:34:11.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:11.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:11.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:11.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:11.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:11.202+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:11.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:11.202+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:11.602 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:11.319+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.320164+0000) 2024-02-19T20:34:11.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:11.519+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.520374+0000) 2024-02-19T20:34:12.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:11.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:12.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:11.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:12.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:11.706+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:12.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:11.707+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:12.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:11.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:12.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:11.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:12.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:11.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:12.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:11.655+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:12.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:11.741+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:11.742215+0000) 2024-02-19T20:34:12.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:12.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:12.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:12.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:12.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:12.243+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:12.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:12.244+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:12.602 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:12.321+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:12.322204+0000) 2024-02-19T20:34:12.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:12.521+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:12.522547+0000) 2024-02-19T20:34:13.014 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:12.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:13.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:12.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:13.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:12.620+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:13.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:12.621+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:13.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:12.743+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:12.744256+0000) 2024-02-19T20:34:13.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:12.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:13.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:12.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:13.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:12.749+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:13.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:12.750+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:13.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:13.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:13.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:13.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:13.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:13.198+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:13.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:13.199+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:13.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:13.323+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.324364+0000) 2024-02-19T20:34:13.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:13.523+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.524675+0000) 2024-02-19T20:34:14.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:13.649+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66873 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:14.015 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:13.645+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.646504+0000) 2024-02-19T20:34:14.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:13.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:14.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:13.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:14.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:13.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:14.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:13.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:14.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:13.645+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.646038+0000) 2024-02-19T20:34:14.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:13.743+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.744883+0000) 2024-02-19T20:34:14.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:13.645+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:13.646379+0000) 2024-02-19T20:34:14.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:13.701+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:14.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:13.701+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:14.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:13.701+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:14.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:13.702+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:14.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:14.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:14.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:14.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:14.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:14.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:14.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:14.193+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:14.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:14.525+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:14.525289+0000) 2024-02-19T20:34:14.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:14.325+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:14.325068+0000) 2024-02-19T20:34:14.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:14.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:14.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:14.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:14.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:14.572+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:14.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:14.573+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:15.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:14.746+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:14.746961+0000) 2024-02-19T20:34:15.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:14.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:15.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:14.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:15.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:14.737+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:15.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:14.738+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:15.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:15.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:15.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:15.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:15.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:15.192+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:15.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:15.192+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:15.602 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:15.326+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:15.327135+0000) 2024-02-19T20:34:15.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:15.527+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:15.528835+0000) 2024-02-19T20:34:15.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:15.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:15.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:15.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:15.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:15.543+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:15.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:15.544+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:16.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:15.748+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:15.749070+0000) 2024-02-19T20:34:16.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:15.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:16.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:15.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:16.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:15.774+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:16.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:15.774+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:16.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:16.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:16.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:16.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:16.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:16.230+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:16.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:16.231+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:16.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:16.328+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.329565+0000) 2024-02-19T20:34:16.571 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:16.530+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.531063+0000) 2024-02-19T20:34:17.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:16.582+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:17.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:16.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:17.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:16.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:17.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:16.583+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:17.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:16.750+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:16.751207+0000) 2024-02-19T20:34:17.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:16.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:17.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:16.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:17.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:16.822+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:17.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:16.822+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:17.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:17.244+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:17.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:17.244+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:17.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:17.244+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:17.343 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:17.245+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:17.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:17.532+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:17.533118+0000) 2024-02-19T20:34:17.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:17.330+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:17.331593+0000) 2024-02-19T20:34:17.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:17.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:17.603 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:17.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:17.604 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:17.566+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:17.604 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:17.567+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:18.015 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:17.752+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:17.753440+0000) 2024-02-19T20:34:18.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:17.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:18.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:17.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:18.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:17.812+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:18.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:17.813+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:18.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:18.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:18.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:18.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:18.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:18.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:18.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:18.242+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:18.600 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:18.332+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.333703+0000) 2024-02-19T20:34:18.600 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:18.534+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.535297+0000) 2024-02-19T20:34:19.015 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:18.657+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66913 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:19.016 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:18.653+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.654348+0000) 2024-02-19T20:34:19.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:18.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:19.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:18.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:19.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:18.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:19.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:18.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:19.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:18.654+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.655302+0000) 2024-02-19T20:34:19.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:18.753+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.754470+0000) 2024-02-19T20:34:19.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:18.653+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:18.654991+0000) 2024-02-19T20:34:19.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:18.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:19.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:18.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:19.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:18.821+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:19.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:18.822+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:19.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:19.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:19.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:19.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:19.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:19.250+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:19.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:19.250+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:19.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:19.535+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:19.536132+0000) 2024-02-19T20:34:19.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:19.333+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:19.334389+0000) 2024-02-19T20:34:20.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:19.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:20.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:19.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:20.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:19.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:20.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:19.840+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:20.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:19.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:20.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:19.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:20.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:19.631+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:20.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:19.632+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:20.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:19.755+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:19.756501+0000) 2024-02-19T20:34:20.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:20.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:20.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:20.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:20.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:20.258+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:20.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:20.259+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:20.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:20.335+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:20.336807+0000) 2024-02-19T20:34:20.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:20.537+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:20.538737+0000) 2024-02-19T20:34:21.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:20.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:21.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:20.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:21.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:20.799+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:21.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:20.800+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:21.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:20.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:21.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:20.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:21.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:20.678+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:21.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:20.679+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:21.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:20.758+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:20.759058+0000) 2024-02-19T20:34:21.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:21.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:21.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:21.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:21.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:21.245+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:21.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:21.246+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:21.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:21.539+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.540742+0000) 2024-02-19T20:34:21.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:21.337+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.338812+0000) 2024-02-19T20:34:22.015 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:21.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:22.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:21.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:22.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:21.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:22.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:21.787+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:22.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:21.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:22.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:21.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:22.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:21.727+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:22.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:21.728+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:22.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:21.760+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:21.761064+0000) 2024-02-19T20:34:22.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:22.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:22.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:22.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:22.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:22.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:22.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:22.242+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:22.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:22.339+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:22.340818+0000) 2024-02-19T20:34:22.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:22.541+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:22.542787+0000) 2024-02-19T20:34:23.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:22.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:23.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:22.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:23.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:22.772+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:23.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:22.773+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:23.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:22.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:23.016 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:22.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:23.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:22.683+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:23.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:22.684+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:23.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:22.762+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:22.763483+0000) 2024-02-19T20:34:23.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:23.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:23.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:23.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:23.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:23.218+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:23.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:23.218+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:23.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:23.544+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.545331+0000) 2024-02-19T20:34:23.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:23.341+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.342894+0000) 2024-02-19T20:34:24.016 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:23.662+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.663339+0000) 2024-02-19T20:34:24.016 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:23.665+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66952 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:24.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:23.661+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.662683+0000) 2024-02-19T20:34:24.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:23.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:24.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:23.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:24.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:23.811+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:24.017 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:23.811+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:24.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:23.662+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.663061+0000) 2024-02-19T20:34:24.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:23.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:24.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:23.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:24.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:23.664+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:24.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:23.665+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:24.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:23.763+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:23.764337+0000) 2024-02-19T20:34:24.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:24.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:24.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:24.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:24.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:24.262+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:24.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:24.262+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:24.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:24.544+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:24.545960+0000) 2024-02-19T20:34:24.603 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:24.342+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:24.343569+0000) 2024-02-19T20:34:25.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:24.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:25.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:24.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:25.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:24.813+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:25.016 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:24.814+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:25.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:24.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:25.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:24.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:25.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:24.675+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:25.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:24.676+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:25.017 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:24.765+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:24.766489+0000) 2024-02-19T20:34:25.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:25.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:25.536 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:25.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:25.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:25.291+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:25.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:25.291+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:25.537 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:25.346+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:25.347066+0000) 2024-02-19T20:34:25.821 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:25.548+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:25.549238+0000) 2024-02-19T20:34:25.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:25.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:25.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:25.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:25.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:25.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:25.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:25.694+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:25.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:25.768+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:25.769946+0000) 2024-02-19T20:34:26.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:25.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:26.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:25.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:26.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:25.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:26.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:25.860+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:26.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:26.280+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:26.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:26.280+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:26.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:26.280+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:26.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:26.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:26.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:26.348+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.349197+0000) 2024-02-19T20:34:26.821 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:26.551+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.552714+0000) 2024-02-19T20:34:26.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:26.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:26.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:26.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:26.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:26.679+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:26.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:26.680+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:26.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:26.772+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:26.773549+0000) 2024-02-19T20:34:27.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:26.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:27.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:26.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:27.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:26.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:27.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:26.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:27.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:27.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:27.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:27.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:27.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:27.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:27.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:27.299+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:27.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:27.350+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:27.351197+0000) 2024-02-19T20:34:27.821 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:27.555+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:27.556371+0000) 2024-02-19T20:34:27.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:27.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:27.821 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:27.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:27.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:27.643+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:27.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:27.644+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:27.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:27.774+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:27.775733+0000) 2024-02-19T20:34:28.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:27.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:28.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:27.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:28.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:27.938+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:28.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:27.938+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:28.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:28.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:28.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:28.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:28.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:28.249+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:28.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:28.250+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:28.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:28.353+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.354184+0000) 2024-02-19T20:34:28.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:28.673+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 66992 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:28.821 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:28.669+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.670415+0000) 2024-02-19T20:34:28.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:28.557+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.558464+0000) 2024-02-19T20:34:28.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:28.670+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.671149+0000) 2024-02-19T20:34:28.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:28.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:28.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:28.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:28.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:28.610+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:28.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:28.611+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:28.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:28.669+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.670887+0000) 2024-02-19T20:34:28.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:28.775+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:28.776412+0000) 2024-02-19T20:34:29.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:28.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:29.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:28.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:29.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:28.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:29.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:28.932+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:29.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:29.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:29.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:29.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:29.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:29.234+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:29.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:29.234+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:29.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:29.353+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:29.354825+0000) 2024-02-19T20:34:29.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:29.558+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:29.559120+0000) 2024-02-19T20:34:29.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:29.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:29.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:29.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:29.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:29.655+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:29.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:29.656+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:30.029 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:29.778+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:29.779763+0000) 2024-02-19T20:34:30.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:29.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:30.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:29.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:30.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:29.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:30.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:29.960+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:30.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:30.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:30.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:30.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:30.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:30.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:30.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:30.261+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:30.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:30.355+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:30.356900+0000) 2024-02-19T20:34:30.821 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:30.561+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:30.562467+0000) 2024-02-19T20:34:30.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:30.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:30.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:30.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:30.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:30.612+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:30.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:30.613+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:30.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:30.780+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:30.781921+0000) 2024-02-19T20:34:31.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:30.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:31.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:30.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:31.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:30.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:31.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:30.998+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:31.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:31.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:31.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:31.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:31.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:31.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:31.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:31.279+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:31.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:31.359+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.360530+0000) 2024-02-19T20:34:31.821 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:31.564+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.565903+0000) 2024-02-19T20:34:31.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:31.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:31.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:31.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:31.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:31.604+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:31.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:31.605+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:31.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:31.782+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:31.783947+0000) 2024-02-19T20:34:32.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:31.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:32.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:31.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:32.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:31.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:32.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:31.957+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:32.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:32.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:32.307+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:32.308+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:32.363+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:32.364036+0000) 2024-02-19T20:34:32.821 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:32.568+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:32.569393+0000) 2024-02-19T20:34:32.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:32.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:32.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:32.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:32.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:32.618+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:32.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:32.618+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:32.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:32.786+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:32.787470+0000) 2024-02-19T20:34:33.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:32.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:33.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:32.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:33.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:32.960+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:33.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:32.960+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:33.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:33.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:33.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:33.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:33.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:33.317+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:33.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:33.318+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:33.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:33.366+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.367494+0000) 2024-02-19T20:34:33.821 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:33.681+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67031 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:33.822 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:33.679+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.680027+0000) 2024-02-19T20:34:33.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:33.570+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.571536+0000) 2024-02-19T20:34:33.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:33.678+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.679484+0000) 2024-02-19T20:34:33.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:33.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:33.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:33.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:33.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:33.623+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:33.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:33.624+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:33.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:33.678+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.679148+0000) 2024-02-19T20:34:33.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:33.787+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:33.788109+0000) 2024-02-19T20:34:34.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:33.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:34.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:33.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:34.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:33.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:34.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:33.959+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:34.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:34.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:34.272+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:34.272+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:34.367+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:34.368521+0000) 2024-02-19T20:34:34.821 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:34.571+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:34.572169+0000) 2024-02-19T20:34:34.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:34.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:34.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:34.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:34.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:34.649+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:34.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:34.649+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:34.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:34.789+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:34.790086+0000) 2024-02-19T20:34:35.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:34.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:35.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:34.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:35.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:34.956+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:35.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:34.957+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:35.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:35.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:35.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:35.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:35.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:35.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:35.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:35.276+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:35.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:35.370+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:35.371930+0000) 2024-02-19T20:34:35.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:35.573+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:35.574204+0000) 2024-02-19T20:34:35.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:35.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:35.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:35.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:35.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:35.629+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:35.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:35.630+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:35.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:35.791+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:35.792082+0000) 2024-02-19T20:34:36.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:35.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:36.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:35.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:36.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:35.966+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:36.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:35.966+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:36.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:36.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:36.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:36.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:36.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:36.283+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:36.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:36.283+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:36.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:36.374+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.375367+0000) 2024-02-19T20:34:36.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:36.575+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.576240+0000) 2024-02-19T20:34:36.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:36.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:36.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:36.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:36.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:36.611+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:36.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:36.612+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:36.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:36.793+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:36.794085+0000) 2024-02-19T20:34:37.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:37.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:37.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:37.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:37.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:37.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:37.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:37.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:37.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:37.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:37.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:37.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:37.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:37.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:37.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:37.321+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:37.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:37.376+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:37.377573+0000) 2024-02-19T20:34:37.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:37.577+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:37.578286+0000) 2024-02-19T20:34:37.822 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:37.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:37.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:37.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:37.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:37.583+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:37.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:37.584+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:37.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:37.795+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:37.796067+0000) 2024-02-19T20:34:38.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:38.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:38.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:38.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:38.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:38.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:38.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:38.003+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:38.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:38.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:38.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:38.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:38.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:38.318+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:38.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:38.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:38.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:38.380+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.381059+0000) 2024-02-19T20:34:38.822 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:38.689+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67071 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:38.823 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:38.686+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.687116+0000) 2024-02-19T20:34:38.823 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:38.579+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.580391+0000) 2024-02-19T20:34:38.823 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:38.685+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.686674+0000) 2024-02-19T20:34:38.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:38.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:38.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:38.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:38.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:38.596+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:38.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:38.596+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:38.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:38.685+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.686869+0000) 2024-02-19T20:34:38.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:38.795+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:38.796711+0000) 2024-02-19T20:34:39.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:39.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:39.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:39.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:39.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:39.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:39.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:39.023+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:39.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:39.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:39.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:39.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:39.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:39.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:39.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:39.321+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:39.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:39.381+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:39.382041+0000) 2024-02-19T20:34:39.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:39.580+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:39.581132+0000) 2024-02-19T20:34:39.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:39.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:39.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:39.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:39.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:39.634+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:39.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:39.635+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:40.045 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:39.798+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:39.799108+0000) 2024-02-19T20:34:40.309 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:40.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:40.309 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:40.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:40.309 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:40.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:40.310 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:40.057+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:40.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:40.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:40.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:40.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:40.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:40.321+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:40.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:40.321+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:40.571 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:40.383+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:40.384689+0000) 2024-02-19T20:34:40.822 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:40.582+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:40.583355+0000) 2024-02-19T20:34:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:40.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:40.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:40.660+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:40.661+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:40.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:40.801+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:40.802776+0000) 2024-02-19T20:34:41.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:41.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:41.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:41.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:41.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:41.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:41.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:41.050+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:41.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:41.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:41.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:41.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:41.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:41.310+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:41.374 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:41.310+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:41.375 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:41.385+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.386785+0000) 2024-02-19T20:34:41.699 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:41.584+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.585906+0000) 2024-02-19T20:34:42.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:41.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:42.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:41.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:42.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:41.711+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:42.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:41.712+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:42.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:41.804+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:41.805440+0000) 2024-02-19T20:34:42.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:42.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:42.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:42.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:42.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:42.070+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:42.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:42.070+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:42.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:42.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:42.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:42.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:42.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:42.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:42.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:42.319+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:42.576 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:42.388+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:42.389372+0000) 2024-02-19T20:34:43.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:42.586+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:42.588004+0000) 2024-02-19T20:34:43.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:42.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:43.018 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:42.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:43.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:42.739+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:43.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:42.740+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:43.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:42.807+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:42.808114+0000) 2024-02-19T20:34:43.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:43.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:43.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:43.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:43.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:43.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:43.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:43.050+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:43.605 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:43.589+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.590541+0000) 2024-02-19T20:34:43.605 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:43.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:43.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:43.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:43.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:43.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:43.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:43.366+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:43.606 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:43.391+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.392179+0000) 2024-02-19T20:34:43.989 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:43.698+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67110 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:43.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:43.693+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.694580+0000) 2024-02-19T20:34:43.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:43.694+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.695084+0000) 2024-02-19T20:34:43.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:44.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:43.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:44.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:43.990 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:44.001+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:43.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:43.694+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.695118+0000) 2024-02-19T20:34:43.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:43.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:43.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:43.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:43.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:43.774+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:43.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:43.775+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:43.991 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:43.807+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:43.808795+0000) 2024-02-19T20:34:44.248 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:44.001+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:44.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:44.375+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:44.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:44.375+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:44.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:44.375+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:44.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:44.375+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:44.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:44.391+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:44.392851+0000) 2024-02-19T20:34:44.823 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:44.590+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:44.591291+0000) 2024-02-19T20:34:44.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:44.810+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:44.811040+0000) 2024-02-19T20:34:44.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:44.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:44.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:44.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:44.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:44.811+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:44.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:44.812+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:45.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:45.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:45.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:45.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.023+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:45.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:45.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:45.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:45.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:45.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:45.371+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:45.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:45.371+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:45.759 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:45.393+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:45.394989+0000) 2024-02-19T20:34:45.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.592+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:45.593380+0000) 2024-02-19T20:34:46.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:46.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:46.018 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.982+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:46.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:45.982+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:46.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:45.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:46.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:45.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:46.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:45.771+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:46.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:45.772+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:46.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:45.812+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:45.813333+0000) 2024-02-19T20:34:46.583 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:46.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:46.583 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:46.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:46.583 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:46.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:46.583 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:46.333+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:46.584 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:46.396+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.397738+0000) 2024-02-19T20:34:46.935 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:46.595+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.596146+0000) 2024-02-19T20:34:46.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:46.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:46.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:46.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:46.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:46.947+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:46.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:46.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:46.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:46.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:46.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:46.748+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:46.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:46.749+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:46.937 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:46.814+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:46.815416+0000) 2024-02-19T20:34:47.252 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:46.948+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:47.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:47.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:47.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:47.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:47.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:47.325+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:47.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:47.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:47.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:47.398+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:47.399778+0000) 2024-02-19T20:34:47.823 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:47.597+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:47.598978+0000) 2024-02-19T20:34:47.823 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:47.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:47.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:47.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:47.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:47.726+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:47.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:47.727+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:47.824 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:47.816+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:47.817488+0000) 2024-02-19T20:34:48.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:47.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:48.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:47.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:48.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:47.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:48.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:47.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:48.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:48.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:48.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:48.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:48.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:48.302+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:48.389 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:48.303+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:48.687 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:48.600+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.601773+0000) 2024-02-19T20:34:48.687 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:48.400+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.401903+0000) 2024-02-19T20:34:48.983 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:48.706+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67150 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:48.983 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:48.701+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.702683+0000) 2024-02-19T20:34:48.984 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:48.702+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.703064+0000) 2024-02-19T20:34:48.984 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:48.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:48.984 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:48.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:48.984 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:48.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:48.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:48.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:48.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:48.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:48.984 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:48.699+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:48.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:48.700+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:48.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:48.703+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.704515+0000) 2024-02-19T20:34:48.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:48.817+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:48.818211+0000) 2024-02-19T20:34:49.253 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:48.995+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:49.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:49.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:49.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:49.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:49.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:49.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:49.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:49.341+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:49.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:49.402+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:49.403154+0000) 2024-02-19T20:34:49.807 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:49.601+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:49.602460+0000) 2024-02-19T20:34:49.807 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:49.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:49.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:49.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:49.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:49.658+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:49.808 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:49.659+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:50.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:49.819+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:49.820219+0000) 2024-02-19T20:34:50.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:50.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:50.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:50.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:50.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:50.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:50.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:50.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:50.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:50.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:50.393 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:50.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:50.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.603+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:50.604441+0000) 2024-02-19T20:34:50.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:50.404+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:50.405185+0000) 2024-02-19T20:34:50.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:50.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:50.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:50.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:50.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:50.615+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:50.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:50.615+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:51.123 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:50.821+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:50.822347+0000) 2024-02-19T20:34:51.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:51.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:51.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.994+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:51.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:50.995+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:51.394 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:51.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:51.394 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:51.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:51.394 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:51.266+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:51.395 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:51.266+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:51.647 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:51.605+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.606478+0000) 2024-02-19T20:34:51.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:51.406+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.407360+0000) 2024-02-19T20:34:52.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:51.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:52.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:51.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:52.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:51.659+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:52.010 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:51.660+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:52.011 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:51.823+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:51.824490+0000) 2024-02-19T20:34:52.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:52.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:52.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:52.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:52.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:52.220+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:52.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:52.220+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:52.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:52.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:52.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:52.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:52.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:52.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:52.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:52.023+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:52.672 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:52.408+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:52.409697+0000) 2024-02-19T20:34:52.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:52.607+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:52.608701+0000) 2024-02-19T20:34:53.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:52.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:53.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:52.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:53.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:52.684+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:53.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:52.685+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:53.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:52.825+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:52.826523+0000) 2024-02-19T20:34:53.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:53.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:53.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:53.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:53.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:53.185+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:53.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:53.185+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:53.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:53.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:53.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:53.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:53.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:53.060+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:53.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:53.060+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:53.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:53.410+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.411749+0000) 2024-02-19T20:34:53.696 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:53.610+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.611258+0000) 2024-02-19T20:34:54.019 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:53.714+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67189 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:54.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:53.712+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.713155+0000) 2024-02-19T20:34:54.020 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:53.711+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.712285+0000) 2024-02-19T20:34:54.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:53.711+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.712153+0000) 2024-02-19T20:34:54.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:53.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:54.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:53.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:54.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:53.716+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:54.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:53.717+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:54.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:53.826+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:53.827631+0000) 2024-02-19T20:34:54.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:54.097+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:54.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:54.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:54.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:54.098+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:54.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:54.098+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:54.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:54.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:54.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:54.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:54.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:54.223+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:54.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:54.223+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:54.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:54.411+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:54.412562+0000) 2024-02-19T20:34:54.682 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:54.611+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:54.612335+0000) 2024-02-19T20:34:55.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:54.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:55.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:54.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:55.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:54.694+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:55.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:54.695+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:55.019 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:54.830+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:54.831060+0000) 2024-02-19T20:34:55.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:55.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:55.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:55.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:55.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:55.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:55.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:55.137+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:55.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:55.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:55.279 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:55.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:55.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:55.255+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:55.280 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:55.256+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:55.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:55.613+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:55.614491+0000) 2024-02-19T20:34:55.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:55.413+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:55.414699+0000) 2024-02-19T20:34:55.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:55.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:55.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:55.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:55.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:55.654+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:55.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:55.655+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:56.076 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:55.833+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:55.834754+0000) 2024-02-19T20:34:56.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:56.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:56.076 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:56.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:56.077 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:56.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:56.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:56.089+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:56.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:56.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:56.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:56.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:56.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:56.208+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:56.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:56.209+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:56.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:56.416+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.417251+0000) 2024-02-19T20:34:56.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:56.615+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.616489+0000) 2024-02-19T20:34:56.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:56.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:56.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:56.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:56.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:56.609+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:56.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:56.610+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:57.076 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:56.837+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:56.838413+0000) 2024-02-19T20:34:57.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:57.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:57.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:57.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:57.408 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:57.241+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:57.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:57.242+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:57.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:57.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:57.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:57.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:57.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:57.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:57.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:57.089+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:57.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:57.617+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:57.618603+0000) 2024-02-19T20:34:57.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:57.418+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:57.419348+0000) 2024-02-19T20:34:57.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:57.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:57.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:57.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:57.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:57.579+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:57.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:57.579+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:58.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:57.840+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:57.841040+0000) 2024-02-19T20:34:58.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:58.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:58.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:58.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:58.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:58.055+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:58.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:58.056+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:58.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:58.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:58.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:58.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:58.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:58.267+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:58.409 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:58.267+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:58.704 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:58.619+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.620753+0000) 2024-02-19T20:34:58.704 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:58.421+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.422084+0000) 2024-02-19T20:34:58.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:58.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:58.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:58.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:58.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:58.536+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:58.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:58.536+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:59.018 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:34:58.723+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67227 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:34:59.019 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:58.719+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.720325+0000) 2024-02-19T20:34:59.019 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:58.719+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.720121+0000) 2024-02-19T20:34:59.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:58.718+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.719938+0000) 2024-02-19T20:34:59.020 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:58.840+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:58.841838+0000) 2024-02-19T20:34:59.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:59.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:59.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:34:59.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.031+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:34:59.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:59.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:59.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:59.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:59.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:59.521+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:34:59.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:59.522+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:59.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:59.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:59.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:59.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:59.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:59.281+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:34:59.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:59.282+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:34:59.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:34:59.422+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:59.423194+0000) 2024-02-19T20:34:59.824 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.620+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:59.621443+0000) 2024-02-19T20:35:00.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:34:59.842+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:34:59.843993+0000) 2024-02-19T20:35:00.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:00.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:00.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.983+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:00.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:34:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:34:59.983+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:00.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:00.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:00.414 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:00.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:00.414 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:00.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:00.414 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:00.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:00.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:00.624+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:00.624936+0000) 2024-02-19T20:35:00.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:00.426+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:00.426617+0000) 2024-02-19T20:35:00.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:00.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:00.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:00.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:00.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:00.556+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:00.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:00.557+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:01.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:00.846+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:00.846098+0000) 2024-02-19T20:35:01.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:00.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:01.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:00.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:01.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:00.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:01.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:00.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:01.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:01.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:01.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:01.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:01.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:01.304+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:01.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:01.304+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:01.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:01.626+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.627564+0000) 2024-02-19T20:35:01.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:01.429+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.429281+0000) 2024-02-19T20:35:01.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:01.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:01.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:01.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:01.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:01.512+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:01.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:01.513+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:02.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:01.847+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:01.848474+0000) 2024-02-19T20:35:02.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:01.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:02.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:01.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:02.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:01.962+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:02.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:01.963+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:02.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:02.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:02.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:02.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:02.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:02.260+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:02.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:02.261+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:02.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:02.431+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:02.432051+0000) 2024-02-19T20:35:02.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:02.629+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:02.630128+0000) 2024-02-19T20:35:02.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:02.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:02.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:02.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:02.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:02.519+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:02.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:02.520+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:03.124 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:02.849+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:02.850511+0000) 2024-02-19T20:35:03.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:02.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:03.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:02.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:03.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:02.935+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:03.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:02.935+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:03.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:03.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:03.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:03.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:03.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:03.300+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:03.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:03.301+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:03.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:03.433+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.434567+0000) 2024-02-19T20:35:03.712 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:03.631+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.632194+0000) 2024-02-19T20:35:03.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:03.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:03.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:03.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:03.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:03.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:03.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:03.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:04.020 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:03.730+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67268 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:04.020 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:03.726+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.727867+0000) 2024-02-19T20:35:04.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:03.727+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.728838+0000) 2024-02-19T20:35:04.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:03.850+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.851255+0000) 2024-02-19T20:35:04.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:03.727+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:03.728268+0000) 2024-02-19T20:35:04.021 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:03.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:04.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:03.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:04.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:03.939+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:04.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:03.940+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:04.270 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:04.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:04.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:04.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:04.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:04.264+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:04.271 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:04.264+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:04.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:04.434+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:04.435847+0000) 2024-02-19T20:35:04.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:04.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:04.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:04.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:04.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:04.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:04.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:04.515+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:04.824 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:04.631+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:04.632904+0000) 2024-02-19T20:35:05.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:04.853+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:04.854625+0000) 2024-02-19T20:35:05.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:04.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:05.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:04.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:05.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:04.913+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:05.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:04.914+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:05.424 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:05.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:05.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:05.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:05.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:05.298+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:05.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:05.298+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:05.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:05.635+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:05.636206+0000) 2024-02-19T20:35:05.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:05.436+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:05.437869+0000) 2024-02-19T20:35:05.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:05.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:05.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:05.470+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:05.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:05.470+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:06.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:05.856+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:05.857187+0000) 2024-02-19T20:35:06.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:05.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:06.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:05.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:06.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:05.955+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:06.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:05.956+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:06.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:06.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:06.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:06.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:06.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:06.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:06.417 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:06.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:06.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:06.439+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.440052+0000) 2024-02-19T20:35:06.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:06.637+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.638347+0000) 2024-02-19T20:35:06.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:06.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:06.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:06.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:06.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:06.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:06.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:06.429+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:07.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:06.858+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:06.859213+0000) 2024-02-19T20:35:07.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:06.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:07.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:06.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:07.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:06.907+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:07.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:06.908+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:07.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:07.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:07.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:07.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:07.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:07.313+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:07.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:07.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:07.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:07.639+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:07.640393+0000) 2024-02-19T20:35:07.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:07.441+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:07.442068+0000) 2024-02-19T20:35:07.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:07.424+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:07.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:07.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:07.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:07.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:07.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:07.425+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:08.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:07.860+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:07.861282+0000) 2024-02-19T20:35:08.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:07.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:08.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:07.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:08.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:07.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:08.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:07.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:08.431 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:08.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:08.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:08.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:08.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:08.279+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:08.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:08.280+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:08.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:08.443+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.444162+0000) 2024-02-19T20:35:08.720 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:08.641+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.642447+0000) 2024-02-19T20:35:08.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:08.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:08.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:08.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:08.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:08.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:08.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:08.456+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:09.021 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:08.739+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67306 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:09.021 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:08.735+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.736578+0000) 2024-02-19T20:35:09.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:08.735+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.736264+0000) 2024-02-19T20:35:09.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:08.861+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.862058+0000) 2024-02-19T20:35:09.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:08.737+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:08.738613+0000) 2024-02-19T20:35:09.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:08.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:09.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:08.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:09.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:08.902+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:09.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:08.902+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:09.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:09.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:09.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:09.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:09.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:09.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:09.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:09.484+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:09.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:09.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:09.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:09.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:09.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:09.316+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:09.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:09.316+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:09.530 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:09.444+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:09.445137+0000) 2024-02-19T20:35:09.825 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:09.642+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:09.643597+0000) 2024-02-19T20:35:10.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:09.864+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:09.865748+0000) 2024-02-19T20:35:10.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:09.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:10.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:09.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:10.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:09.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:10.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:09.884+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:10.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:10.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:10.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:10.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:10.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:10.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:10.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:10.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:10.434 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:10.446+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:10.447177+0000) 2024-02-19T20:35:10.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:10.644+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:10.645778+0000) 2024-02-19T20:35:10.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:10.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:10.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:10.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:10.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:10.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:10.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:10.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:11.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:10.867+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:10.868240+0000) 2024-02-19T20:35:11.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:10.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:11.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:10.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:11.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:10.914+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:11.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:10.915+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:11.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:11.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:11.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:11.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:11.436 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:11.294+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:11.437 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:11.295+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:11.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:11.448+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.449545+0000) 2024-02-19T20:35:11.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:11.647+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.648177+0000) 2024-02-19T20:35:11.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:11.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:11.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:11.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:11.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:11.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:11.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:11.468+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:12.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:11.869+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:11.870802+0000) 2024-02-19T20:35:12.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:11.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:12.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:11.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:12.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:11.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:12.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:11.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:12.413 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:12.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:12.414 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:12.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:12.414 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:12.333+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:12.414 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:12.334+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:12.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:12.650+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:12.651828+0000) 2024-02-19T20:35:12.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:12.450+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:12.451985+0000) 2024-02-19T20:35:12.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:12.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:12.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:12.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:12.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:12.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:12.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:12.426+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:13.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:12.872+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:12.873292+0000) 2024-02-19T20:35:13.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:12.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:13.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:12.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:13.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:12.899+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:13.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:12.900+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:13.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:13.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:13.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:13.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:13.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:13.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:13.377 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:13.344+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:13.641 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:13.453+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.454137+0000) 2024-02-19T20:35:13.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:13.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:13.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:13.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:13.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:13.389+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:13.641 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:13.390+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:13.925 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:13.743+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.744166+0000) 2024-02-19T20:35:13.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:13.743+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.744522+0000) 2024-02-19T20:35:13.925 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:13.872+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.873977+0000) 2024-02-19T20:35:13.926 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:13.747+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67347 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:13.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:13.653+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.654057+0000) 2024-02-19T20:35:13.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:13.744+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:13.745380+0000) 2024-02-19T20:35:13.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:13.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:13.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:13.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:13.926 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:13.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:14.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:13.938+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:14.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:14.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:14.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:14.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:14.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:14.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:14.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:14.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:14.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:14.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:14.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:14.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:14.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:14.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:14.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:14.353+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:14.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:14.454+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:14.455358+0000) 2024-02-19T20:35:14.826 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:14.654+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:14.655010+0000) 2024-02-19T20:35:15.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:14.874+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:14.875954+0000) 2024-02-19T20:35:15.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:14.932+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:15.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:14.932+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:15.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:14.932+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:15.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:14.933+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:15.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:15.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:15.377 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:15.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:15.378 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:15.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:15.378 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:15.333+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:15.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:15.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:15.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:15.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:15.378 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:15.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:15.379 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:15.390+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:15.645 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:15.456+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:15.457796+0000) 2024-02-19T20:35:15.931 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:15.878+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:15.879385+0000) 2024-02-19T20:35:15.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:15.657+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:15.658573+0000) 2024-02-19T20:35:15.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:15.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:15.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:15.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:15.932 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:15.943+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:16.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:15.944+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:16.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:16.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:16.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:16.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:16.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:16.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:16.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:16.379+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:16.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:16.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:16.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:16.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:16.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:16.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:16.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:16.404+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:16.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:16.460+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.461277+0000) 2024-02-19T20:35:16.826 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:16.661+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.662059+0000) 2024-02-19T20:35:17.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:16.880+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:16.881426+0000) 2024-02-19T20:35:17.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:16.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:17.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:16.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:17.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:16.937+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:17.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:16.938+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:17.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:17.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:17.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:17.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:17.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:17.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:17.395 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:17.350+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:17.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:17.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:17.652 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:17.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:17.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:17.407+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:17.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:17.407+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:17.653 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:17.463+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:17.464816+0000) 2024-02-19T20:35:17.911 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:17.884+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:17.885014+0000) 2024-02-19T20:35:17.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:17.664+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:17.665555+0000) 2024-02-19T20:35:17.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:17.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:17.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:17.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:17.912 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:17.923+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:18.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:17.924+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:18.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:18.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:18.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:18.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:18.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:18.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:18.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:18.320+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:18.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:18.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:18.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:18.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:18.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:18.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:18.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:18.372+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:18.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:18.465+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.466859+0000) 2024-02-19T20:35:18.826 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:18.752+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.753162+0000) 2024-02-19T20:35:18.826 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:18.751+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.752231+0000) 2024-02-19T20:35:18.827 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:18.755+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67385 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:18.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:18.666+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.667847+0000) 2024-02-19T20:35:18.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:18.752+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.753209+0000) 2024-02-19T20:35:19.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:18.884+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:18.885779+0000) 2024-02-19T20:35:19.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:18.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:19.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:18.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:19.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:18.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:19.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:18.895+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:19.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:19.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:19.399 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:19.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:19.399 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:19.319+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:19.399 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:19.320+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:19.655 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:19.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:19.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:19.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:19.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:19.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:19.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:19.411+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:19.656 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:19.466+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:19.467626+0000) 2024-02-19T20:35:20.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:19.886+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:19.887876+0000) 2024-02-19T20:35:20.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:19.667+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:19.668726+0000) 2024-02-19T20:35:20.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:19.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:20.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:19.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:20.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:19.886+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:20.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:19.887+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:20.559 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:20.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:20.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:20.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:20.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:20.349+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:20.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:20.350+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:20.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:20.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:20.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:20.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:20.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:20.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:20.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:20.440+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:20.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:20.468+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:20.469617+0000) 2024-02-19T20:35:20.826 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:20.669+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:20.670963+0000) 2024-02-19T20:35:21.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:20.890+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:20.891320+0000) 2024-02-19T20:35:21.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:20.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:21.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:20.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:21.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:20.917+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:21.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:20.918+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:21.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:21.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:21.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:21.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:21.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:21.391+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:21.660 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:21.392+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:21.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:21.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:21.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:21.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:21.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:21.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:21.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:21.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:21.661 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:21.472+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.473280+0000) 2024-02-19T20:35:22.021 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:21.893+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.894827+0000) 2024-02-19T20:35:22.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:21.672+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:21.673086+0000) 2024-02-19T20:35:22.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:21.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:22.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:21.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:22.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:21.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:22.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:21.870+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:22.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:22.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:22.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:22.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:22.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:22.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:22.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:22.411+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:22.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:22.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:22.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:22.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:22.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:22.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:22.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:22.440+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:22.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:22.474+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:22.475232+0000) 2024-02-19T20:35:22.826 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:22.674+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:22.675420+0000) 2024-02-19T20:35:23.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:22.897+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:22.898403+0000) 2024-02-19T20:35:23.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:22.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:23.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:22.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:23.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:22.920+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:23.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:22.920+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:23.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:23.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:23.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:23.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:23.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:23.417+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:23.666 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:23.418+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:23.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:23.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:23.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:23.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:23.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:23.472+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:23.667 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:23.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:23.668 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:23.477+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.478709+0000) 2024-02-19T20:35:24.022 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:23.762+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67426 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:24.022 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:23.759+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.760307+0000) 2024-02-19T20:35:24.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:23.759+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.760687+0000) 2024-02-19T20:35:24.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:23.898+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.899132+0000) 2024-02-19T20:35:24.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:23.677+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.678872+0000) 2024-02-19T20:35:24.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:23.760+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:23.761263+0000) 2024-02-19T20:35:24.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:23.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:24.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:23.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:24.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:23.874+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:24.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:23.875+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:24.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:24.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:24.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:24.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:24.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:24.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:24.560 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:24.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:24.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:24.478+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:24.479864+0000) 2024-02-19T20:35:24.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:24.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:24.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:24.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:24.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:24.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:24.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:24.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:24.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:24.679+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:24.680036+0000) 2024-02-19T20:35:25.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:24.901+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:24.902772+0000) 2024-02-19T20:35:25.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:24.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:25.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:24.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:25.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:24.842+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:25.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:24.843+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:25.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:25.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:25.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:25.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:25.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:25.443+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:25.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:25.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:25.669 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:25.482+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:25.483353+0000) 2024-02-19T20:35:25.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:25.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:25.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:25.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:25.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:25.396+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:25.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:25.397+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:26.022 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:25.903+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:25.904825+0000) 2024-02-19T20:35:26.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:25.681+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:25.682165+0000) 2024-02-19T20:35:26.022 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:25.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:26.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:25.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:26.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:25.853+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:26.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:25.854+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:26.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:26.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:26.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:26.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:26.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:26.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:26.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:26.411+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:26.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:26.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:26.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:26.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:26.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:26.422+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:26.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:26.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:26.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:26.485+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.486860+0000) 2024-02-19T20:35:26.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:26.684+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.685542+0000) 2024-02-19T20:35:27.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:26.906+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:26.907358+0000) 2024-02-19T20:35:27.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:26.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:27.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:26.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:27.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:26.878+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:27.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:26.879+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:27.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:27.686+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:27.687585+0000) 2024-02-19T20:35:27.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:27.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:27.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:27.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:27.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:27.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:27.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:27.453+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:27.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:27.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:27.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:27.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:27.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:27.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:27.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:27.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:27.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:27.489+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:27.490345+0000) 2024-02-19T20:35:28.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:27.909+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:27.910926+0000) 2024-02-19T20:35:28.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:27.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:28.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:27.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:28.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:27.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:28.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:27.846+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:28.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:28.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:28.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:28.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:28.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:28.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:28.679 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:28.412+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:28.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:28.492+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.493883+0000) 2024-02-19T20:35:28.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:28.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:28.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:28.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:28.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:28.496+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:28.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:28.497+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:29.022 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:28.771+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67464 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:29.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:28.766+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.767944+0000) 2024-02-19T20:35:29.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:28.767+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.768814+0000) 2024-02-19T20:35:29.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:28.911+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.912158+0000) 2024-02-19T20:35:29.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:28.689+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.690210+0000) 2024-02-19T20:35:29.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:28.767+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:28.768168+0000) 2024-02-19T20:35:29.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:28.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:29.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:28.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:29.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:28.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:29.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:28.892+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:29.560 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:29.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:29.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:29.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:29.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:29.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:29.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:29.460+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:29.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:29.494+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:29.495042+0000) 2024-02-19T20:35:29.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:29.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:29.561 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:29.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:29.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:29.502+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:29.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:29.503+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:29.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:29.690+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:29.691054+0000) 2024-02-19T20:35:30.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:29.914+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:29.915117+0000) 2024-02-19T20:35:30.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:29.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:30.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:29.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:30.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:29.883+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:30.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:29.883+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:30.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:30.692+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:30.693721+0000) 2024-02-19T20:35:30.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:30.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:30.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:30.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:30.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:30.495+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:30.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:30.496+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:30.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:30.497+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:30.498630+0000) 2024-02-19T20:35:30.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:30.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:30.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:30.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:30.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:30.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:30.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:30.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:31.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:30.917+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:30.918723+0000) 2024-02-19T20:35:31.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:30.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:31.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:30.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:31.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:30.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:31.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:30.882+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:31.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:31.694+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.695843+0000) 2024-02-19T20:35:31.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:31.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:31.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:31.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:31.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:31.474+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:31.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:31.474+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:31.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:31.501+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.502257+0000) 2024-02-19T20:35:31.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:31.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:31.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:31.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:31.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:31.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:31.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:31.455+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:32.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:31.920+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:31.921249+0000) 2024-02-19T20:35:32.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:31.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:32.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:31.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:32.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:31.918+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:32.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:31.919+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:32.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:32.697+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:32.698059+0000) 2024-02-19T20:35:32.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:32.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:32.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:32.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:32.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:32.463+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:32.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:32.463+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:32.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:32.505+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:32.506098+0000) 2024-02-19T20:35:32.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:32.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:32.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:32.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:32.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:32.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:32.780 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:32.503+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:33.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:32.922+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:32.923771+0000) 2024-02-19T20:35:33.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:32.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:33.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:32.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:33.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:32.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:33.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:32.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:33.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:33.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:33.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:33.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:33.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:33.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:33.761 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:33.468+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:33.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:33.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:33.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:33.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:33.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:33.469+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:33.761 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:33.469+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:33.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:33.507+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.508207+0000) 2024-02-19T20:35:33.762 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:33.700+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.701626+0000) 2024-02-19T20:35:34.023 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:33.779+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67505 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:34.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:33.775+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.776665+0000) 2024-02-19T20:35:34.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:33.775+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.776485+0000) 2024-02-19T20:35:34.024 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:33.923+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.924562+0000) 2024-02-19T20:35:34.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:33.776+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:33.777067+0000) 2024-02-19T20:35:34.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:33.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:34.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:33.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:34.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:33.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:34.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:33.860+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:34.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:34.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:34.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:34.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:34.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:34.425+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:34.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:34.425+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:34.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:34.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:34.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:34.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:34.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:34.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:34.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:34.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:34.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:34.508+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:34.509086+0000) 2024-02-19T20:35:34.827 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:34.701+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:34.702432+0000) 2024-02-19T20:35:35.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:34.925+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:34.926603+0000) 2024-02-19T20:35:35.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:34.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:35.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:34.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:35.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:34.857+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:35.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:34.858+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:35.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:35.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:35.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:35.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:35.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:35.451+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:35.691 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:35.452+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:35.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:35.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:35.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:35.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:35.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:35.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:35.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:35.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:35.692 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:35.510+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:35.511107+0000) 2024-02-19T20:35:36.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:35.927+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:35.928647+0000) 2024-02-19T20:35:36.023 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:35.703+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:35.704455+0000) 2024-02-19T20:35:36.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:35.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:36.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:35.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:36.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:35.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:36.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:35.883+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:36.561 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:36.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:36.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:36.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:36.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:36.463+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:36.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:36.464+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:36.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:36.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:36.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:36.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:36.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:36.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:36.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:36.461+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:36.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:36.512+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.513137+0000) 2024-02-19T20:35:36.828 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:36.705+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.706710+0000) 2024-02-19T20:35:37.127 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:36.929+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:36.930852+0000) 2024-02-19T20:35:37.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:36.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:37.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:36.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:37.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:36.891+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:37.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:36.892+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:37.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:37.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:37.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:37.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:37.697 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:37.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:37.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:37.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:37.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:37.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:37.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:37.424+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:37.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:37.425+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:37.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:37.514+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:37.515240+0000) 2024-02-19T20:35:38.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:37.933+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:37.934352+0000) 2024-02-19T20:35:38.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:37.709+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:37.710088+0000) 2024-02-19T20:35:38.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:37.933+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:38.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:37.933+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:38.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:37.933+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:38.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:37.934+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:38.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:38.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:38.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:38.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:38.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:38.400+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:38.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:38.401+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:38.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:38.516+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.517269+0000) 2024-02-19T20:35:38.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:38.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:38.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:38.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:38.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:38.471+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:38.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:38.472+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:38.828 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:38.787+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67543 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:38.828 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:38.783+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.784750+0000) 2024-02-19T20:35:38.829 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:38.784+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.785220+0000) 2024-02-19T20:35:38.829 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:38.711+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.712340+0000) 2024-02-19T20:35:38.829 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:38.783+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.784532+0000) 2024-02-19T20:35:39.128 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:38.934+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:38.935576+0000) 2024-02-19T20:35:39.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:38.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:39.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:38.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:39.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:38.950+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:39.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:38.950+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:39.700 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:39.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:39.700 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:39.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:39.700 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:39.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:39.700 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:39.508+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:39.700 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:39.449+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:39.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:39.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:39.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:39.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:39.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:39.450+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:39.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:39.517+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:39.518523+0000) 2024-02-19T20:35:39.986 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:39.936+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:39.937666+0000) 2024-02-19T20:35:39.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:39.712+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:39.713102+0000) 2024-02-19T20:35:39.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:39.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:39.987 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:39.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:40.268 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:39.999+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:40.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:39.999+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:40.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:40.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:40.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:40.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:40.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:40.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:40.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:40.475+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:40.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:40.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:40.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:40.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:40.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:40.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:40.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:40.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:40.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:40.519+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:40.520988+0000) 2024-02-19T20:35:40.828 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:40.714+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:40.715754+0000) 2024-02-19T20:35:41.128 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:40.938+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:40.939709+0000) 2024-02-19T20:35:41.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:41.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:41.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:41.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:41.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:41.012+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:41.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:41.012+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:41.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:41.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:41.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:41.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:41.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:41.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:41.705 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:41.462+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:41.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:41.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:41.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:41.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:41.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:41.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:41.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:41.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:41.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:41.522+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.523631+0000) 2024-02-19T20:35:41.993 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:41.940+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.941720+0000) 2024-02-19T20:35:41.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:41.717+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:41.718354+0000) 2024-02-19T20:35:41.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:41.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:41.994 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:42.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:42.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:42.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:42.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:42.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:42.562 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:42.453+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:42.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:42.454+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:42.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:42.525+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:42.526228+0000) 2024-02-19T20:35:42.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:42.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:42.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:42.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:42.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:42.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:42.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:42.473+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:42.828 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.720+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:42.721063+0000) 2024-02-19T20:35:43.128 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:42.942+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:42.943742+0000) 2024-02-19T20:35:43.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:43.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:43.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.964+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:43.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:42.965+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:43.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:43.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:43.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:43.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:43.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:43.445+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:43.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:43.445+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:43.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:43.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:43.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:43.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:43.711 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:43.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:43.711 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:43.428+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:43.711 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:43.527+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.528772+0000) 2024-02-19T20:35:43.984 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:43.795+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67584 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:43.985 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:43.792+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.793144+0000) 2024-02-19T20:35:43.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:43.792+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.793236+0000) 2024-02-19T20:35:43.985 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:43.943+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.944657+0000) 2024-02-19T20:35:43.985 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:43.722+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.723181+0000) 2024-02-19T20:35:43.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:43.791+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:43.792982+0000) 2024-02-19T20:35:43.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:43.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:43.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:43.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:43.986 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:43.996+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:44.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:43.997+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:44.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:44.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:44.562 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:44.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:44.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:44.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:44.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:44.419+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:44.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:44.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:44.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:44.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:44.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:44.441+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:44.563 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:44.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:44.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:44.528+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:44.529710+0000) 2024-02-19T20:35:44.828 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:44.723+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:44.724298+0000) 2024-02-19T20:35:45.128 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:44.946+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:44.947067+0000) 2024-02-19T20:35:45.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:44.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:45.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:44.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:45.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:44.990+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:45.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:44.990+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:45.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:45.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:45.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:45.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:45.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:45.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:45.714 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:45.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:45.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:45.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:45.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:45.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:45.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:45.473+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:45.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:45.473+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:45.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:45.530+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:45.531825+0000) 2024-02-19T20:35:46.013 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:45.948+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:45.949048+0000) 2024-02-19T20:35:46.013 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:45.725+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:45.726713+0000) 2024-02-19T20:35:46.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:46.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:46.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:46.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:46.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:46.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:46.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:46.026+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:46.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:46.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:46.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:46.449+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:46.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:46.449+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:46.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:46.449+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:46.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:46.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:46.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:46.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:46.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:46.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:46.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:46.514+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:46.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:46.532+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.533892+0000) 2024-02-19T20:35:46.939 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:46.728+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.729013+0000) 2024-02-19T20:35:47.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:46.951+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:46.952611+0000) 2024-02-19T20:35:47.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:47.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:47.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:47.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:47.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:47.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:47.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:47.016+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:47.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:47.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:47.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:47.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:47.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:47.457+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:47.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:47.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:47.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:47.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:47.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:47.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:47.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:47.512+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:47.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:47.513+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:47.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:47.535+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:47.535984+0000) 2024-02-19T20:35:47.941 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:47.731+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:47.731035+0000) 2024-02-19T20:35:48.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:47.955+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:47.955160+0000) 2024-02-19T20:35:48.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:48.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:48.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:48.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:48.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:48.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:48.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:48.064+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:48.612 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:48.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:48.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:48.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:48.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:48.429+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:48.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:48.429+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:48.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:48.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:48.613 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:48.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:48.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:48.467+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:48.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:48.468+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:48.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:48.537+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.538084+0000) 2024-02-19T20:35:48.910 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:48.800+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.801777+0000) 2024-02-19T20:35:48.910 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:48.732+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.733131+0000) 2024-02-19T20:35:48.911 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:48.800+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.801375+0000) 2024-02-19T20:35:48.911 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:48.800+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.801209+0000) 2024-02-19T20:35:48.911 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:48.803+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67622 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:49.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:48.954+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:48.955971+0000) 2024-02-19T20:35:49.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.017+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:49.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:49.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:49.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.018+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:49.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:49.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:49.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:49.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:49.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:49.443+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:49.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:49.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:49.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:49.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:49.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:49.426+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:49.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:49.426+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:49.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:49.537+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:49.538919+0000) 2024-02-19T20:35:49.944 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.733+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:49.734056+0000) 2024-02-19T20:35:50.269 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:49.957+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:49.958399+0000) 2024-02-19T20:35:50.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:50.269 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:50.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.978+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:50.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:49.978+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:50.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:50.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:50.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:50.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:50.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:50.427+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:50.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:50.427+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:50.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:50.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:50.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:50.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:50.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:50.413+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:50.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:50.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:50.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:50.540+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:50.541153+0000) 2024-02-19T20:35:50.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:50.735+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:50.736079+0000) 2024-02-19T20:35:51.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:50.959+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:50.960964+0000) 2024-02-19T20:35:51.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:51.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:51.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:51.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:51.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:51.020+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:51.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:51.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:51.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:51.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:51.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:51.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:51.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:51.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:51.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:51.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:51.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:51.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:51.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:51.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:51.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:51.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:51.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:51.409+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:51.726 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:51.542+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.543502+0000) 2024-02-19T20:35:51.995 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:51.962+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.963537+0000) 2024-02-19T20:35:51.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:51.737+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:51.738356+0000) 2024-02-19T20:35:51.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:51.995 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:51.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.007+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:52.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:52.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:52.386+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:52.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:52.386+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:52.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:52.386+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:52.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:52.387+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:52.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:52.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:52.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:52.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:52.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:52.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:52.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:52.361+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:52.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:52.545+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:52.546182+0000) 2024-02-19T20:35:52.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.739+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:52.741000+0000) 2024-02-19T20:35:53.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:52.964+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:52.965550+0000) 2024-02-19T20:35:53.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:53.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:53.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.988+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:53.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:52.989+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:53.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:53.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:53.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:53.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:53.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:53.366+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:53.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:53.366+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:53.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:53.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:53.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:53.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:53.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:53.336+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:53.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:53.337+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:53.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:53.547+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.548726+0000) 2024-02-19T20:35:53.915 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:53.811+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67663 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:53.915 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:53.808+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.809989+0000) 2024-02-19T20:35:53.916 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:53.742+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.743690+0000) 2024-02-19T20:35:53.916 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:53.808+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.809501+0000) 2024-02-19T20:35:53.916 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:53.808+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.809352+0000) 2024-02-19T20:35:54.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:53.965+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:53.966440+0000) 2024-02-19T20:35:54.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:53.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:54.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:53.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:54.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:53.985+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:54.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:53.986+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:54.613 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:54.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:54.639 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:54.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:54.639 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:54.342+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:54.640 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:54.343+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:54.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:54.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:54.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:54.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:54.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:54.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:54.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:54.342+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:54.640 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:54.548+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:54.549570+0000) 2024-02-19T20:35:54.955 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:54.743+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:54.744965+0000) 2024-02-19T20:35:55.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:54.967+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:54.968457+0000) 2024-02-19T20:35:55.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:55.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:55.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:55.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:55.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:55.028+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:55.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:55.029+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:55.563 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:55.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:55.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:55.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:55.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:55.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:55.564 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:55.312+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:55.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:55.564 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:55.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:55.314+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:55.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:55.314+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:55.565 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:55.550+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:55.551560+0000) 2024-02-19T20:35:55.830 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:55.746+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:55.747060+0000) 2024-02-19T20:35:56.130 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:55.969+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:55.970666+0000) 2024-02-19T20:35:56.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:56.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:56.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:56.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:56.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:56.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:56.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:56.021+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:56.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:56.339+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:56.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:56.339+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:56.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:56.339+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:56.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:56.340+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:56.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:56.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:56.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:56.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:56.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:56.342+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:56.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:56.343+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:56.830 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:56.552+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.553568+0000) 2024-02-19T20:35:56.830 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:56.748+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.749777+0000) 2024-02-19T20:35:57.130 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:56.971+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:56.972840+0000) 2024-02-19T20:35:57.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:57.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:57.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:57.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:57.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:57.021+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:57.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:57.022+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:57.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:57.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:57.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:57.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:57.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:57.383+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:57.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:57.384+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:57.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:57.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:57.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:57.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:57.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:57.324+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:57.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:57.325+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:57.830 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:57.751+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:57.752345+0000) 2024-02-19T20:35:57.830 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:57.554+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:57.555634+0000) 2024-02-19T20:35:58.130 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:57.974+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:57.975042+0000) 2024-02-19T20:35:58.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:58.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:58.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:58.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:58.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:58.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:58.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:58.049+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:58.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:58.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:58.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:58.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:58.419 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:58.373+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:58.420 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:58.373+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:58.741 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:58.557+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.558241+0000) 2024-02-19T20:35:58.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:58.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:58.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:58.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:58.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:58.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:58.741 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:58.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:59.025 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:35:58.819+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67702 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:35:59.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:58.815+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.816828+0000) 2024-02-19T20:35:59.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:58.816+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.817128+0000) 2024-02-19T20:35:59.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:58.974+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.975970+0000) 2024-02-19T20:35:59.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:58.753+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.754556+0000) 2024-02-19T20:35:59.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:58.816+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:58.817481+0000) 2024-02-19T20:35:59.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:59.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:59.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:59.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:59.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:59.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:35:59.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:59.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:35:59.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:59.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:59.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:59.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:59.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:59.360+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:35:59.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:59.360+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:59.614 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:35:59.558+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:59.559152+0000) 2024-02-19T20:35:59.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:59.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:59.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:59.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:59.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:59.442+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:35:59.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:59.443+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:35:59.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:35:59.754+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:59.755533+0000) 2024-02-19T20:36:00.270 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:35:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:35:59.978+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:35:59.979400+0000) 2024-02-19T20:36:00.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:00.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:00.270 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:00.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:00.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:00.048+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:00.271 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:00.049+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:00.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:00.361+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:00.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:00.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:00.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:00.362+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:00.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:00.362+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:00.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:00.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:00.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:00.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:00.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:00.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:00.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:00.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:00.830 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:00.757+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:00.758180+0000) 2024-02-19T20:36:00.831 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:00.560+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:00.561739+0000) 2024-02-19T20:36:01.084 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:00.981+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:00.982047+0000) 2024-02-19T20:36:01.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:01.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:01.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:01.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:01.454 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:01.328+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:01.455 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:01.328+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:01.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:01.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:01.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:01.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:01.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:01.096+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:01.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:01.097+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:01.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:01.562+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.563777+0000) 2024-02-19T20:36:01.747 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:01.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:01.747 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:01.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:01.747 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:01.467+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:01.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:01.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:02.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:01.983+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.984042+0000) 2024-02-19T20:36:02.026 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:01.759+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:01.760722+0000) 2024-02-19T20:36:02.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:02.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:02.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:02.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:02.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:02.071+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:02.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:02.072+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:02.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:02.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:02.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:02.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:02.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:02.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:02.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:02.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:02.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:02.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:02.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:02.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:02.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:02.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:02.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:02.365+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:02.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:02.565+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:02.566362+0000) 2024-02-19T20:36:02.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:02.762+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:02.763300+0000) 2024-02-19T20:36:03.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:02.985+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:02.986469+0000) 2024-02-19T20:36:03.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:03.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:03.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:03.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:03.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:03.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:03.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:03.043+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:03.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:03.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:03.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:03.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:03.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:03.390+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:03.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:03.391+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:03.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:03.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:03.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:03.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:03.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:03.330+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:03.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:03.331+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:03.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:03.568+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.569839+0000) 2024-02-19T20:36:03.935 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:03.828+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67743 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:03.935 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:03.824+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.825163+0000) 2024-02-19T20:36:03.936 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:03.824+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.825564+0000) 2024-02-19T20:36:03.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:03.765+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.766904+0000) 2024-02-19T20:36:03.936 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:03.823+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.824614+0000) 2024-02-19T20:36:04.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:03.986+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:03.987650+0000) 2024-02-19T20:36:04.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:04.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:04.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:04.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:04.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:04.052+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:04.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:04.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:04.557 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:04.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:04.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:04.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:04.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:04.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:04.558 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:04.405+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:04.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:04.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:04.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:04.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:04.558 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:04.303+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:04.559 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:04.303+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:04.831 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:04.570+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:04.571203+0000) 2024-02-19T20:36:04.831 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:04.766+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:04.767760+0000) 2024-02-19T20:36:05.130 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:04.988+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:04.989679+0000) 2024-02-19T20:36:05.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:05.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:05.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:05.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:05.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:05.027+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:05.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:05.028+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:05.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:05.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:05.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:05.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:05.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:05.353+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:05.407 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:05.353+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:05.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:05.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:05.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:05.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:05.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:05.419+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:05.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:05.420+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:05.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:05.572+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:05.573396+0000) 2024-02-19T20:36:06.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:05.990+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:05.991665+0000) 2024-02-19T20:36:06.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:05.768+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:05.769798+0000) 2024-02-19T20:36:06.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:06.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:06.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:06.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:06.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:06.010+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:06.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:06.011+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:06.614 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:06.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:06.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:06.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:06.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:06.427+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:06.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:06.428+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:06.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:06.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:06.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:06.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:06.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:06.365+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:06.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:06.366+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:06.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:06.574+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.576004+0000) 2024-02-19T20:36:06.980 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:06.772+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.773251+0000) 2024-02-19T20:36:07.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:06.992+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:06.993728+0000) 2024-02-19T20:36:07.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:07.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:07.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:07.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:07.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:07.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:07.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:07.013+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:07.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:07.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:07.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:07.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:07.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:07.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:07.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:07.439+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:07.615 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:07.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:07.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:07.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:07.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:07.370+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:07.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:07.371+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:07.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:07.577+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:07.578069+0000) 2024-02-19T20:36:07.983 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:07.775+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:07.776816+0000) 2024-02-19T20:36:08.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:07.996+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:07.997294+0000) 2024-02-19T20:36:08.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:08.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:08.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:08.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:08.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:08.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:08.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:08.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:08.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:08.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:08.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:08.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:08.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:08.421+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:08.565 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:08.422+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:08.566 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:08.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:08.566 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:08.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:08.566 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:08.410+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:08.566 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:08.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:08.816 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:08.777+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.778925+0000) 2024-02-19T20:36:08.817 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:08.579+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.580674+0000) 2024-02-19T20:36:09.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:08.831+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.832946+0000) 2024-02-19T20:36:09.131 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:08.836+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67781 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:09.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:08.832+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.833334+0000) 2024-02-19T20:36:09.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:08.997+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.998504+0000) 2024-02-19T20:36:09.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:08.832+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:08.833714+0000) 2024-02-19T20:36:09.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:09.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:09.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.009+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:09.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.010+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:09.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:09.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:09.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:09.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:09.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:09.377+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:09.406 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:09.378+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:09.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:09.580+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:09.581578+0000) 2024-02-19T20:36:09.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:09.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:09.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:09.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:09.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:09.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:09.767 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:09.419+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:10.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:09.999+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:10.000748+0000) 2024-02-19T20:36:10.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.778+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:09.779889+0000) 2024-02-19T20:36:10.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:10.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:10.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.975+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:10.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:09.976+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:10.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:10.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:10.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:10.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:10.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:10.444+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:10.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:10.445+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:10.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:10.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:10.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:10.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:10.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:10.379+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:10.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:10.380+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:10.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:10.584+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:10.585155+0000) 2024-02-19T20:36:10.989 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:10.781+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:10.782046+0000) 2024-02-19T20:36:11.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:11.001+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.002866+0000) 2024-02-19T20:36:11.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:11.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:11.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:11.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:11.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:11.016+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:11.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:11.017+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:11.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:11.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:11.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:11.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:11.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:11.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:11.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:11.454+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:11.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:11.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:11.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:11.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:11.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:11.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:11.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:11.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:11.618 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:11.586+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.587164+0000) 2024-02-19T20:36:11.991 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:11.783+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:11.784067+0000) 2024-02-19T20:36:12.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:12.004+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:12.005131+0000) 2024-02-19T20:36:12.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:12.066+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:12.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:12.066+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:12.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:12.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:12.280 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:12.067+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:12.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:12.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:12.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:12.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:12.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:12.486+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:12.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:12.487+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:12.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:12.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:12.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:12.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:12.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:12.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:12.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:12.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:12.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:12.589+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:12.590878+0000) 2024-02-19T20:36:12.993 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:12.785+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:12.786120+0000) 2024-02-19T20:36:13.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:13.006+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.007261+0000) 2024-02-19T20:36:13.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:13.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:13.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:13.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:13.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:13.057+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:13.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:13.059+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:13.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:13.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:13.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:13.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:13.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:13.503+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:13.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:13.504+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:13.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:13.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:13.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:13.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:13.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:13.372+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:13.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:13.372+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:13.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:13.592+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.593506+0000) 2024-02-19T20:36:13.950 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:13.844+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67822 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:13.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:13.840+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.841275+0000) 2024-02-19T20:36:13.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:13.839+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.840764+0000) 2024-02-19T20:36:13.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:13.787+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.788553+0000) 2024-02-19T20:36:13.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:13.840+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:13.841691+0000) 2024-02-19T20:36:14.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:14.007+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:14.008314+0000) 2024-02-19T20:36:14.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:14.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:14.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:14.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:14.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:14.063+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:14.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:14.064+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:14.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:14.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:14.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:14.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:14.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:14.494+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:14.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:14.495+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:14.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:14.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:14.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:14.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:14.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:14.347+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:14.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:14.347+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:14.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:14.593+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:14.594397+0000) 2024-02-19T20:36:14.996 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:14.788+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:14.789410+0000) 2024-02-19T20:36:15.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:15.009+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:15.010340+0000) 2024-02-19T20:36:15.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:15.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:15.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:15.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:15.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:15.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:15.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:15.091+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:15.583 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:15.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:15.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:15.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:15.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:15.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:15.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:15.480+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:15.584 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:15.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:15.584 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:15.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:15.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:15.343+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:15.585 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:15.344+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:15.998 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:15.790+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:15.791999+0000) 2024-02-19T20:36:15.999 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:15.596+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:15.597057+0000) 2024-02-19T20:36:16.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:16.011+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.012404+0000) 2024-02-19T20:36:16.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:16.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:16.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:16.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:16.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:16.085+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:16.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:16.086+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:16.615 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:16.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:16.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:16.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:16.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:16.508+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:16.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:16.509+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:16.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:16.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:16.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:16.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:16.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:16.394+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:16.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:16.394+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:16.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:16.599+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.600685+0000) 2024-02-19T20:36:17.000 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:16.793+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:16.794623+0000) 2024-02-19T20:36:17.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:17.013+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:17.014540+0000) 2024-02-19T20:36:17.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:17.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:17.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:17.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:17.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:17.105+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:17.279 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:17.105+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:17.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:17.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:17.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:17.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:17.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:17.492+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:17.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:17.493+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:17.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:17.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:17.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:17.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:17.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:17.404+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:17.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:17.405+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:17.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:17.603+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:17.604551+0000) 2024-02-19T20:36:18.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:17.795+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:17.796876+0000) 2024-02-19T20:36:18.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:18.015+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.016849+0000) 2024-02-19T20:36:18.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:18.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:18.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:18.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:18.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:18.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:18.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:18.084+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:18.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:18.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:18.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:18.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:18.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:18.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:18.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:18.461+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:18.616 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:18.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:18.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:18.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:18.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:18.415+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:18.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:18.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:18.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:18.606+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.607100+0000) 2024-02-19T20:36:18.959 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:18.852+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67860 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:18.959 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:18.847+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.848586+0000) 2024-02-19T20:36:18.960 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:18.847+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.848422+0000) 2024-02-19T20:36:18.960 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:18.799+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.800718+0000) 2024-02-19T20:36:18.960 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:18.849+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:18.850497+0000) 2024-02-19T20:36:19.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:19.017+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:19.018070+0000) 2024-02-19T20:36:19.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:19.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:19.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:19.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:19.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:19.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:19.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:19.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:19.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:19.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:19.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:19.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:19.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:19.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:19.616 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:19.474+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:19.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:19.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:19.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:19.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:19.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:19.428+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:19.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:19.429+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:19.617 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:19.607+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:19.608316+0000) 2024-02-19T20:36:20.007 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:19.801+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:19.802150+0000) 2024-02-19T20:36:20.272 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:20.019+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:20.020715+0000) 2024-02-19T20:36:20.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:20.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:20.272 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:20.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:20.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:20.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:20.273 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:20.034+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:20.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:20.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:20.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:20.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:20.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:20.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:20.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:20.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:20.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:20.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:20.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:20.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:20.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:20.421+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:20.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:20.422+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:20.833 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:20.803+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:20.804206+0000) 2024-02-19T20:36:20.833 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:20.609+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:20.610902+0000) 2024-02-19T20:36:21.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:21.022+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.023289+0000) 2024-02-19T20:36:21.132 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:21.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:21.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:21.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:21.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:21.022+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:21.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:21.023+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:21.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:21.375+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:21.442 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:21.375+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:21.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:21.375+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:21.443 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:21.376+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:21.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:21.611+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.612944+0000) 2024-02-19T20:36:21.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:21.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:21.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:21.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:21.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:21.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:21.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:21.456+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:22.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:22.025+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:22.026707+0000) 2024-02-19T20:36:22.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:21.806+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:21.807848+0000) 2024-02-19T20:36:22.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:22.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:22.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:22.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:22.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:22.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:22.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:22.024+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:22.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:22.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:22.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:22.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:22.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:22.401+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:22.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:22.402+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:22.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:22.614+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:22.615014+0000) 2024-02-19T20:36:22.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:22.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:22.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:22.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:22.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:22.465+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:22.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:22.465+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:23.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:23.029+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.030334+0000) 2024-02-19T20:36:23.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:22.810+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:22.811504+0000) 2024-02-19T20:36:23.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:23.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:23.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:23.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:23.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:23.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:23.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:23.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:23.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:23.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:23.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:23.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:23.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:23.364+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:23.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:23.365+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:23.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:23.617+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.618500+0000) 2024-02-19T20:36:23.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:23.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:23.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:23.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:23.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:23.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:23.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:23.438+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:24.132 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:23.860+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67901 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:24.133 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:23.857+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.858195+0000) 2024-02-19T20:36:24.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:23.857+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.858496+0000) 2024-02-19T20:36:24.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:24.030+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:24.031674+0000) 2024-02-19T20:36:24.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:23.812+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.813754+0000) 2024-02-19T20:36:24.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:23.857+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:23.858113+0000) 2024-02-19T20:36:24.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:24.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:24.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:24.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:24.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:24.003+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:24.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:24.004+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:24.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:24.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:24.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:24.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:24.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:24.341+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:24.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:24.341+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:24.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:24.618+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:24.619894+0000) 2024-02-19T20:36:24.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:24.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:24.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:24.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:24.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:24.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:24.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:24.435+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:25.132 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:25.034+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:25.035101+0000) 2024-02-19T20:36:25.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:24.813+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:24.814637+0000) 2024-02-19T20:36:25.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:25.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:25.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:25.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:25.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:25.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:25.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:25.008+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:25.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:25.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:25.425 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:25.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:25.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:25.381+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:25.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:25.382+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:25.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:25.620+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:25.621912+0000) 2024-02-19T20:36:25.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:25.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:25.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:25.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:25.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:25.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:25.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:25.439+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:26.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:26.036+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.037067+0000) 2024-02-19T20:36:26.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:25.815+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:25.816763+0000) 2024-02-19T20:36:26.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:26.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:26.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:26.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:26.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:26.030+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:26.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:26.031+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:26.397 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:26.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:26.398 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:26.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:26.398 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:26.386+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:26.398 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:26.387+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:26.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:26.623+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.624384+0000) 2024-02-19T20:36:26.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:26.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:26.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:26.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:26.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:26.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:26.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:26.411+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:27.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:27.038+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:27.039128+0000) 2024-02-19T20:36:27.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:26.818+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:26.819349+0000) 2024-02-19T20:36:27.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:27.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:27.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:27.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:27.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:27.029+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:27.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:27.030+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:27.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:27.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:27.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:27.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:27.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:27.344+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:27.404 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:27.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:27.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:27.626+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:27.627990+0000) 2024-02-19T20:36:27.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:27.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:27.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:27.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:27.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:27.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:27.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:27.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:28.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:28.040+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.041278+0000) 2024-02-19T20:36:28.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:27.820+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:27.821426+0000) 2024-02-19T20:36:28.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:28.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:28.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:28.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:28.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:28.049+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:28.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:28.050+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:28.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:28.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:28.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:28.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:28.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:28.374+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:28.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:28.374+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:28.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:28.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:28.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:28.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:28.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:28.387+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:28.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:28.387+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:28.834 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:28.822+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.823978+0000) 2024-02-19T20:36:28.834 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:28.629+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.630055+0000) 2024-02-19T20:36:29.133 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:28.869+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67939 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:29.133 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:28.865+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.866930+0000) 2024-02-19T20:36:29.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:28.865+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.866148+0000) 2024-02-19T20:36:29.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:29.041+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:29.042257+0000) 2024-02-19T20:36:29.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:28.864+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:28.865497+0000) 2024-02-19T20:36:29.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:29.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:29.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:29.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:29.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:29.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:29.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:29.095+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:29.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:29.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:29.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:29.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:29.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:29.356+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:29.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:29.357+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:29.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:29.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:29.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:29.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:29.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:29.430+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:29.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:29.431+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:29.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:29.629+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:29.630939+0000) 2024-02-19T20:36:30.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:30.043+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:30.044275+0000) 2024-02-19T20:36:30.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:29.824+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:29.825273+0000) 2024-02-19T20:36:30.133 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:30.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:30.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:30.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:30.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:30.065+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:30.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:30.066+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:30.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:30.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:30.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:30.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:30.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:30.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:30.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:30.313+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:30.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:30.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:30.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:30.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:30.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:30.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:30.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:30.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:30.660 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:30.632+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:30.633037+0000) 2024-02-19T20:36:31.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:30.826+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:30.827269+0000) 2024-02-19T20:36:31.297 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:31.045+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.046270+0000) 2024-02-19T20:36:31.297 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:31.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:31.297 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:31.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:31.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:31.064+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:31.298 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:31.065+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:31.567 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:31.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:31.567 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:31.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:31.568 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:31.389+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:31.568 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:31.390+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:31.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:31.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:31.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:31.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:31.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:31.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:31.568 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:31.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:31.834 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:31.633+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.635010+0000) 2024-02-19T20:36:31.834 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:31.828+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:31.829248+0000) 2024-02-19T20:36:32.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:32.048+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:32.049742+0000) 2024-02-19T20:36:32.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:32.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:32.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:32.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:32.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:32.059+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:32.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:32.059+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:32.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:32.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:32.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:32.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:32.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:32.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:32.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:32.295+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:32.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:32.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:32.345+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:32.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:32.345+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:32.834 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:32.830+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:32.831306+0000) 2024-02-19T20:36:32.835 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:32.636+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:32.637038+0000) 2024-02-19T20:36:33.090 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:33.052+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.053267+0000) 2024-02-19T20:36:33.365 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:33.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:33.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:33.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:33.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:33.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:33.366 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:33.104+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:33.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:33.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:33.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:33.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:33.367 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:33.292+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:33.367 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:33.292+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:33.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:33.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:33.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:33.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:33.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:33.349+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:33.367 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:33.351+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:33.870 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:33.879+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 67980 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:33.871 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:33.875+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.875666+0000) 2024-02-19T20:36:33.871 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:33.833+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.833420+0000) 2024-02-19T20:36:33.871 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:33.874+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.874452+0000) 2024-02-19T20:36:33.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:33.639+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.639091+0000) 2024-02-19T20:36:33.872 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:33.874+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:33.874764+0000) 2024-02-19T20:36:34.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:34.054+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:34.054465+0000) 2024-02-19T20:36:34.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:34.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:34.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:34.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:34.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:34.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:34.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:34.063+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:34.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:34.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:34.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:34.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:34.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:34.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:34.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:34.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:34.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:34.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:34.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:34.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:34.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:34.820 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:34.638+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:34.639967+0000) 2024-02-19T20:36:35.133 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:35.055+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:35.056458+0000) 2024-02-19T20:36:35.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:34.833+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:34.834447+0000) 2024-02-19T20:36:35.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:35.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:35.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:35.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:35.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:35.040+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:35.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:35.040+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:35.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:35.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:35.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:35.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:35.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:35.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:35.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:35.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:35.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:35.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:35.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:35.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:35.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:35.403+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:35.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:35.404+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:35.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:35.642+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:35.643427+0000) 2024-02-19T20:36:36.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:36.058+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.059938+0000) 2024-02-19T20:36:36.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:35.835+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:35.836570+0000) 2024-02-19T20:36:36.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:36.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:36.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:36.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:36.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:36.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:36.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:36.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:36.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:36.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:36.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:36.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:36.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:36.384+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:36.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:36.385+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:36.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:36.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:36.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:36.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:36.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:36.260+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:36.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:36.261+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:36.826 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:36.644+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.645544+0000) 2024-02-19T20:36:37.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:37.060+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:37.062002+0000) 2024-02-19T20:36:37.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:36.838+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:36.839884+0000) 2024-02-19T20:36:37.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:37.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:37.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:37.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:37.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:37.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:37.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:37.014+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:37.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:37.230+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:37.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:37.230+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:37.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:37.230+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:37.400 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:37.231+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:37.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:37.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:37.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:37.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:37.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:37.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:37.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:37.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:37.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:37.646+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:37.647581+0000) 2024-02-19T20:36:38.135 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:38.064+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.065514+0000) 2024-02-19T20:36:38.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:37.841+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:37.842077+0000) 2024-02-19T20:36:38.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:38.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:38.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:38.137 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.005+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:38.424 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:38.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:38.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:38.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:38.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:38.267+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:38.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:38.267+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:38.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:38.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:38.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:38.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:38.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:38.437+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:38.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:38.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:38.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:38.648+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.649675+0000) 2024-02-19T20:36:39.134 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:38.886+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68018 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:39.134 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:38.882+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.883959+0000) 2024-02-19T20:36:39.135 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:38.882+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.883400+0000) 2024-02-19T20:36:39.135 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:39.065+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:39.066929+0000) 2024-02-19T20:36:39.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.844+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.845599+0000) 2024-02-19T20:36:39.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.882+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:38.883432+0000) 2024-02-19T20:36:39.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:39.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:39.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:39.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:38.998+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:39.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:39.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:39.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:39.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:39.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:39.314+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:39.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:39.315+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:39.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:39.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:39.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:39.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:39.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:39.416+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:39.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:39.416+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:39.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:39.649+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:39.650545+0000) 2024-02-19T20:36:40.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:40.069+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:40.070383+0000) 2024-02-19T20:36:40.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:39.846+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:39.847061+0000) 2024-02-19T20:36:40.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:39.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:40.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:39.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:40.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:39.991+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:40.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:39.992+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:40.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:40.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:40.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:40.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:40.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:40.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:40.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:40.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:40.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:40.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:40.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:40.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:40.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:40.409+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:40.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:40.410+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:40.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:40.653+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:40.654058+0000) 2024-02-19T20:36:41.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:41.071+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.072677+0000) 2024-02-19T20:36:41.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:40.848+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:40.849094+0000) 2024-02-19T20:36:41.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:41.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:41.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:41.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:41.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:41.018+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:41.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:41.019+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:41.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:41.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:41.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:41.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:41.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:41.395+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:41.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:41.395+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:41.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:41.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:41.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:41.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:41.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:41.350+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:41.529 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:41.351+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:41.835 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:41.655+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.656875+0000) 2024-02-19T20:36:42.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:42.073+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:42.074777+0000) 2024-02-19T20:36:42.134 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:41.850+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:41.851166+0000) 2024-02-19T20:36:42.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:42.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:42.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:42.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:42.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:42.004+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:42.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:42.005+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:42.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:42.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:42.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:42.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:42.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:42.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:42.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:42.369+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:42.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:42.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:42.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:42.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:42.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:42.432+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:42.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:42.433+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:42.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:42.657+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:42.658947+0000) 2024-02-19T20:36:43.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:43.075+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.076863+0000) 2024-02-19T20:36:43.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:42.852+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:42.853297+0000) 2024-02-19T20:36:43.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:43.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:43.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:43.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:43.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:43.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:43.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:43.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:43.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:43.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:43.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:43.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:43.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:43.397+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:43.385 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:43.398+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:43.647 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:43.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:43.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:43.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:43.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:43.446+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:43.648 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:43.446+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:43.898 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:43.890+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.891788+0000) 2024-02-19T20:36:43.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:43.660+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.661601+0000) 2024-02-19T20:36:43.898 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:43.891+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.892716+0000) 2024-02-19T20:36:43.899 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:43.894+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68059 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:43.899 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:43.854+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.855480+0000) 2024-02-19T20:36:43.899 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:43.890+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:43.891723+0000) 2024-02-19T20:36:44.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:44.076+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:44.077808+0000) 2024-02-19T20:36:44.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:44.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:44.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:44.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:44.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:44.036+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:44.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:44.037+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:44.618 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:44.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:44.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:44.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:44.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:44.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:44.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:44.378+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:44.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:44.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:44.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:44.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:44.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:44.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:44.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:44.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:45.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:44.661+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:44.662812+0000) 2024-02-19T20:36:45.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:44.855+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:44.856463+0000) 2024-02-19T20:36:45.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:45.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:45.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.013+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:45.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.014+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:45.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:45.078+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:45.079863+0000) 2024-02-19T20:36:45.618 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:45.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:45.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:45.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:45.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:45.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:45.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:45.360+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:45.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:45.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:45.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:45.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:45.619 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:45.444+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:45.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:45.444+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:45.954 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.858+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:45.859937+0000) 2024-02-19T20:36:45.954 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:45.665+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:45.666261+0000) 2024-02-19T20:36:46.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:46.081+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.082078+0000) 2024-02-19T20:36:46.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:46.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:46.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.967+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:46.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:45.967+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:46.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:46.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:46.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:46.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:46.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:46.330+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:46.569 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:46.331+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:46.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:46.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:46.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:46.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:46.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:46.413+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:46.570 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:46.413+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:46.835 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:46.667+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.668329+0000) 2024-02-19T20:36:47.134 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:47.084+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:47.085596+0000) 2024-02-19T20:36:47.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:46.862+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:46.863223+0000) 2024-02-19T20:36:47.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:46.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:47.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:46.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:47.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:46.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:47.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:46.959+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:47.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:47.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:47.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:47.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:47.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:47.321+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:47.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:47.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:47.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:47.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:47.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:47.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:47.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:47.397+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:47.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:47.397+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:47.836 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:47.669+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:47.670432+0000) 2024-02-19T20:36:48.135 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:48.088+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.089077+0000) 2024-02-19T20:36:48.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:47.865+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:47.866603+0000) 2024-02-19T20:36:48.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:48.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:48.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:48.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:48.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:48.002+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:48.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:48.002+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:48.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:48.275+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:48.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:48.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:48.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:48.276+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:48.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:48.276+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:48.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:48.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:48.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:48.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:48.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:48.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:48.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:48.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:48.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:48.671+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.672491+0000) 2024-02-19T20:36:49.135 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:48.903+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68097 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:49.135 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:48.899+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.900476+0000) 2024-02-19T20:36:49.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:48.899+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.900628+0000) 2024-02-19T20:36:49.136 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:49.089+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:49.090570+0000) 2024-02-19T20:36:49.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:48.869+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.870096+0000) 2024-02-19T20:36:49.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:48.898+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:48.899214+0000) 2024-02-19T20:36:49.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:49.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:49.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.011+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:49.137 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.012+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:49.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:49.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:49.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:49.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:49.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:49.317+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:49.457 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:49.318+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:49.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:49.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:49.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:49.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:49.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:49.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:49.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:49.469+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:49.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:49.672+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:49.673367+0000) 2024-02-19T20:36:50.135 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:50.091+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:50.092660+0000) 2024-02-19T20:36:50.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.870+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:49.871573+0000) 2024-02-19T20:36:50.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:50.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:50.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.998+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:50.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:49.998+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:50.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:50.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:50.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:50.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:50.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:50.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:50.482 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:50.287+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:50.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:50.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:50.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:50.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:50.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:50.495+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:50.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:50.495+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:50.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:50.674+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:50.675405+0000) 2024-02-19T20:36:51.135 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:51.095+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.096231+0000) 2024-02-19T20:36:51.135 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:50.872+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:50.873723+0000) 2024-02-19T20:36:51.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:51.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:51.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.005+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:51.136 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.005+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:51.386 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:51.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:51.386 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:51.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:51.386 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:51.318+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:51.387 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:51.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:51.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:51.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:51.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:51.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:51.663 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:51.460+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:51.664 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:51.460+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:51.944 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:51.676+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.677422+0000) 2024-02-19T20:36:51.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.874+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:51.875843+0000) 2024-02-19T20:36:51.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:51.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:51.945 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.957+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:52.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:51.958+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:52.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:52.098+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:52.099818+0000) 2024-02-19T20:36:52.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:52.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:52.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:52.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:52.619 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:52.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:52.620 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:52.359+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:52.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:52.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:52.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:52.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:52.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:52.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:52.620 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:52.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:52.931 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:52.877+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:52.878087+0000) 2024-02-19T20:36:52.932 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:52.679+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:52.680961+0000) 2024-02-19T20:36:53.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:53.100+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.101900+0000) 2024-02-19T20:36:53.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:52.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:53.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:52.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:53.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:52.944+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:53.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:52.945+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:53.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:53.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:53.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:53.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:53.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:53.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:53.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:53.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:53.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:53.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:53.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:53.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:53.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:53.517+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:53.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:53.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:53.670 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:53.682+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.683050+0000) 2024-02-19T20:36:53.964 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:53.911+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68138 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:53.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:53.879+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.880128+0000) 2024-02-19T20:36:53.965 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:53.907+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.908454+0000) 2024-02-19T20:36:53.965 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:53.907+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.908909+0000) 2024-02-19T20:36:53.965 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:53.907+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:53.908723+0000) 2024-02-19T20:36:54.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:54.101+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:54.102816+0000) 2024-02-19T20:36:54.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:53.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:54.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:53.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:54.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:53.977+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:54.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:53.978+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:54.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:54.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:54.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:54.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:54.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:54.430+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:54.670 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:54.431+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:54.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:54.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:54.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:54.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:54.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:54.554+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:54.671 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:54.554+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:55.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:54.683+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:54.684081+0000) 2024-02-19T20:36:55.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:54.880+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:54.881039+0000) 2024-02-19T20:36:55.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:54.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:55.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:54.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:55.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:54.929+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:55.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:54.930+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:55.388 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:55.104+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:55.105426+0000) 2024-02-19T20:36:55.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:55.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:55.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:55.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:55.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:55.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:55.674 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:55.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:55.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:55.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:55.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:55.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:55.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:55.438+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:55.674 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:55.439+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:56.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:55.686+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:55.687577+0000) 2024-02-19T20:36:56.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:55.882+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:55.883623+0000) 2024-02-19T20:36:56.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:55.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:56.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:55.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:56.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:55.887+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:56.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:55.888+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:56.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:56.106+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.107657+0000) 2024-02-19T20:36:56.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:56.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:56.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:56.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:56.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:56.531+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:56.677 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:56.532+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:56.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:56.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:56.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:56.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:56.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:56.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:56.678 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:56.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:57.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:56.690+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.691139+0000) 2024-02-19T20:36:57.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:56.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:57.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:56.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:57.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:56.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:57.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:56.881+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:57.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:56.885+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:56.886231+0000) 2024-02-19T20:36:57.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:57.110+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:57.111302+0000) 2024-02-19T20:36:57.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:57.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:57.681 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:57.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:57.681 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:57.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:57.681 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:57.460+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:57.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:57.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:57.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:57.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:57.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:57.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:57.682 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:57.510+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:58.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:57.693+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:57.694503+0000) 2024-02-19T20:36:58.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:57.887+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:57.888750+0000) 2024-02-19T20:36:58.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:57.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:58.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:57.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:58.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:57.895+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:58.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:57.896+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:58.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:58.112+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.113756+0000) 2024-02-19T20:36:58.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:58.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:58.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:58.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:58.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:58.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:58.684 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:58.476+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:58.684 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:58.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:58.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:58.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:58.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:58.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:58.685 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:58.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:59.023 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:36:58.919+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68176 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:36:59.023 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:58.915+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.916960+0000) 2024-02-19T20:36:59.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:58.696+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.697896+0000) 2024-02-19T20:36:59.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:58.916+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.917248+0000) 2024-02-19T20:36:59.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:58.891+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.892086+0000) 2024-02-19T20:36:59.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:58.915+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:58.916494+0000) 2024-02-19T20:36:59.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:58.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:59.024 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:58.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:59.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:58.931+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:36:59.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:58.932+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:36:59.279 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:59.114+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:59.115186+0000) 2024-02-19T20:36:59.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:59.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:59.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:59.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:59.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:59.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:36:59.685 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:36:59.440+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:59.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:59.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:59.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:59.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:59.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:59.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:36:59.686 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:59.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:36:59.968 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:36:59.698+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:59.699375+0000) 2024-02-19T20:36:59.968 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:59.892+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:36:59.893142+0000) 2024-02-19T20:37:00.274 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:00.117+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:00.118761+0000) 2024-02-19T20:37:00.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:59.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:00.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:59.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:00.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:59.981+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:00.274 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:36:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:36:59.982+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:00.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:00.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:00.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:00.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:00.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:00.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:00.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:00.460+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:00.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:00.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:00.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:00.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:00.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:00.526+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:00.529 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:00.527+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:00.837 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:00.700+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:00.701603+0000) 2024-02-19T20:37:01.107 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:00.894+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:00.895301+0000) 2024-02-19T20:37:01.107 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:00.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:01.107 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:00.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:01.107 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:00.992+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:01.108 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:00.992+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:01.379 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:01.120+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.121277+0000) 2024-02-19T20:37:01.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:01.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:01.689 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:01.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:01.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:01.450+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:01.690 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:01.451+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:01.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:01.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:01.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:01.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:01.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:01.494+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:01.690 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:01.495+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:01.945 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:01.702+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.703663+0000) 2024-02-19T20:37:01.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:01.897+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:01.898181+0000) 2024-02-19T20:37:01.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:01.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:01.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:01.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:01.946 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:01.959+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:02.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:02.122+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:02.123815+0000) 2024-02-19T20:37:02.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:01.960+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:02.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:02.478+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:02.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:02.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:02.692 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:02.479+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:02.693 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:02.479+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:02.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:02.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:02.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:02.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:02.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:02.532+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:02.693 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:02.532+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:02.974 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:02.705+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:02.706301+0000) 2024-02-19T20:37:02.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:02.900+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:02.901778+0000) 2024-02-19T20:37:02.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:02.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:02.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:02.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:02.975 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:02.987+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:03.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:03.124+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.125850+0000) 2024-02-19T20:37:03.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:02.988+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:03.694 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:03.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:03.695 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:03.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:03.695 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:03.517+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:03.695 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:03.517+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:03.695 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:03.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:03.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:03.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:03.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:03.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:03.696 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:03.570+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:03.950 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:03.927+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68217 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:03.950 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:03.923+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.924576+0000) 2024-02-19T20:37:03.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:03.902+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.903967+0000) 2024-02-19T20:37:03.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:03.923+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.924519+0000) 2024-02-19T20:37:03.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:03.707+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.708797+0000) 2024-02-19T20:37:03.951 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:03.923+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:03.924333+0000) 2024-02-19T20:37:04.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:04.125+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:04.126772+0000) 2024-02-19T20:37:04.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:03.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:04.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:03.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:04.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:03.963+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:04.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:03.964+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:04.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:04.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:04.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:04.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:04.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:04.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:04.696 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:04.475+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:04.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:04.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:04.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:04.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:04.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:04.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:04.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:04.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:04.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:04.709+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:04.710283+0000) 2024-02-19T20:37:04.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:04.903+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:04.904923+0000) 2024-02-19T20:37:04.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:04.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:04.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:04.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:04.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:04.984+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:05.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:05.127+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:05.128754+0000) 2024-02-19T20:37:05.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:04.985+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:05.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:05.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:05.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:05.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:05.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:05.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:05.699 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:05.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:05.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:05.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:05.699 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:05.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:05.700 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:05.505+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:05.700 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:05.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:06.012 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:05.905+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:05.906962+0000) 2024-02-19T20:37:06.013 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:05.711+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:05.712832+0000) 2024-02-19T20:37:06.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:06.131+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.132237+0000) 2024-02-19T20:37:06.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:06.025+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:06.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:06.026+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:06.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:06.026+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:06.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:06.026+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:06.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:06.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:06.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:06.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:06.701 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:06.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:06.702 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:06.462+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:06.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:06.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:06.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:06.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:06.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:06.514+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:06.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:06.515+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:07.002 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:06.714+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.715474+0000) 2024-02-19T20:37:07.002 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:06.907+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:06.908967+0000) 2024-02-19T20:37:07.002 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:07.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:07.002 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:07.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:07.003 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:07.015+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:07.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:07.016+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:07.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:07.133+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:07.134409+0000) 2024-02-19T20:37:07.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:07.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:07.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:07.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:07.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:07.435+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:07.704 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:07.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:07.705 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:07.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:07.705 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:07.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:07.705 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:07.468+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:07.705 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:07.468+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:08.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:07.717+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:07.718067+0000) 2024-02-19T20:37:08.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:07.911+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:07.912452+0000) 2024-02-19T20:37:08.399 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:08.136+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.137066+0000) 2024-02-19T20:37:08.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:08.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:08.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:08.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:08.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:08.053+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:08.400 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:08.053+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:08.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:08.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:08.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:08.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:08.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:08.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:08.706 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:08.413+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:08.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:08.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:08.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:08.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:08.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:08.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:08.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:08.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:09.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:08.931+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.932451+0000) 2024-02-19T20:37:09.028 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:08.935+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68255 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:09.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:08.914+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.915937+0000) 2024-02-19T20:37:09.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:08.931+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.932449+0000) 2024-02-19T20:37:09.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:08.719+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.720181+0000) 2024-02-19T20:37:09.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:08.931+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:08.932277+0000) 2024-02-19T20:37:09.397 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:09.137+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:09.138084+0000) 2024-02-19T20:37:09.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:09.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:09.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:09.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:09.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:09.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:09.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:09.076+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:09.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:09.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:09.709 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:09.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:09.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:09.410+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:09.710 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:09.411+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:09.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:09.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:09.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:09.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:09.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:09.505+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:09.710 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:09.505+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:10.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:09.916+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:09.917469+0000) 2024-02-19T20:37:10.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:09.722+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:09.723528+0000) 2024-02-19T20:37:10.391 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:10.140+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:10.141808+0000) 2024-02-19T20:37:10.392 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:10.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:10.392 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:10.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:10.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:10.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:10.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:10.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:10.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:10.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:10.393 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:10.076+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:10.711 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:10.404+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:10.712 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:10.405+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:10.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:10.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:10.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:10.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:10.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:10.479+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:10.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:10.479+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:11.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:10.918+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:10.919967+0000) 2024-02-19T20:37:11.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:10.724+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:10.726013+0000) 2024-02-19T20:37:11.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:11.144+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.145369+0000) 2024-02-19T20:37:11.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:11.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:11.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:11.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:11.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:11.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:11.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:11.095+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:11.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:11.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:11.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:11.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:11.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:11.436+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:11.715 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:11.436+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:11.715 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:11.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:11.716 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:11.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:11.716 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:11.493+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:11.716 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:11.493+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:12.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:11.922+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.923647+0000) 2024-02-19T20:37:12.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:11.728+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:11.729334+0000) 2024-02-19T20:37:12.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:12.146+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:12.147450+0000) 2024-02-19T20:37:12.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:12.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:12.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:12.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:12.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:12.132+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:12.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:12.133+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:12.717 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:12.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:12.717 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:12.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:12.717 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:12.477+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:12.718 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:12.477+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:12.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:12.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:12.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:12.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:12.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:12.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:12.718 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:12.474+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:13.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:12.926+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:12.927381+0000) 2024-02-19T20:37:13.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:12.730+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:12.731626+0000) 2024-02-19T20:37:13.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:13.148+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.149772+0000) 2024-02-19T20:37:13.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:13.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:13.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:13.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:13.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:13.171+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:13.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:13.172+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:13.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:13.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:13.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:13.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:13.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:13.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:13.720 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:13.442+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:13.721 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:13.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:13.721 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:13.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:13.721 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:13.450+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:13.721 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:13.451+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:14.027 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:13.943+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68296 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:14.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:13.941+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.942176+0000) 2024-02-19T20:37:14.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:13.928+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.929979+0000) 2024-02-19T20:37:14.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:13.939+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.940409+0000) 2024-02-19T20:37:14.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:13.733+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.734254+0000) 2024-02-19T20:37:14.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:13.939+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:13.940547+0000) 2024-02-19T20:37:14.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:14.149+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:14.150873+0000) 2024-02-19T20:37:14.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:14.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:14.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:14.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:14.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:14.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:14.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:14.204+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:14.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:14.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:14.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:14.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:14.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:14.484+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:14.721 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:14.485+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:14.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:14.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:14.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:14.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:14.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:14.438+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:14.722 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:14.438+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:15.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:14.734+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:14.735455+0000) 2024-02-19T20:37:15.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:14.929+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:14.930922+0000) 2024-02-19T20:37:15.408 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:15.152+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:15.153610+0000) 2024-02-19T20:37:15.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:15.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:15.408 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:15.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:15.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:15.229+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:15.409 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:15.230+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:15.724 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:15.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:15.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:15.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:15.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:15.454+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:15.725 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:15.455+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:15.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:15.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:15.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:15.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:15.725 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:15.511+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:15.726 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:15.512+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:16.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:15.738+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:15.739083+0000) 2024-02-19T20:37:16.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:15.931+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:15.932980+0000) 2024-02-19T20:37:16.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:16.154+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.155543+0000) 2024-02-19T20:37:16.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:16.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:16.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:16.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:16.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:16.221+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:16.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:16.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:16.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:16.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:16.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:16.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:16.727 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:16.509+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:16.728 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:16.510+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:16.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:16.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:16.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:16.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:16.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:16.486+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:16.728 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:16.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:17.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:16.934+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.935009+0000) 2024-02-19T20:37:17.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:16.740+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:16.741699+0000) 2024-02-19T20:37:17.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:17.156+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:17.157594+0000) 2024-02-19T20:37:17.410 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:17.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:17.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:17.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:17.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:17.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:17.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:17.196+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:17.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:17.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:17.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:17.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:17.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:17.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:17.730 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:17.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:17.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:17.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:17.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:17.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:17.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:17.458+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:17.731 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:17.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:18.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:17.743+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:17.744296+0000) 2024-02-19T20:37:18.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:17.936+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:17.937063+0000) 2024-02-19T20:37:18.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:18.158+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.159621+0000) 2024-02-19T20:37:18.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:18.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:18.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:18.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:18.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:18.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:18.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:18.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:18.732 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:18.482+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:18.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:18.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:18.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:18.483+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:18.733 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:18.483+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:18.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:18.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:18.733 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:18.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:18.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:18.459+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:18.734 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:18.459+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:19.027 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:18.947+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.948900+0000) 2024-02-19T20:37:19.028 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:18.951+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68334 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:19.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:18.745+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.746901+0000) 2024-02-19T20:37:19.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:18.948+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.949745+0000) 2024-02-19T20:37:19.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:18.939+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.939101+0000) 2024-02-19T20:37:19.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:18.947+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:18.948505+0000) 2024-02-19T20:37:19.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:19.159+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:19.160569+0000) 2024-02-19T20:37:19.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:19.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:19.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:19.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:19.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:19.218+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:19.411 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:19.219+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:19.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:19.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:19.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:19.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:19.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:19.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:19.734 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:19.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:19.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:19.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:19.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:19.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:19.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:19.429+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:19.735 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:19.430+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:20.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:19.939+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:19.940063+0000) 2024-02-19T20:37:20.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:19.748+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:19.748495+0000) 2024-02-19T20:37:20.369 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:20.162+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:20.162558+0000) 2024-02-19T20:37:20.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:20.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:20.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:20.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:20.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:20.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:20.369 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:20.204+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:20.622 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:20.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:20.622 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:20.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:20.622 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:20.475+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:20.622 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:20.476+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:20.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:20.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:20.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:20.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:20.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:20.383+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:20.623 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:20.383+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:20.927 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:20.751+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:20.751137+0000) 2024-02-19T20:37:21.217 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:21.165+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.166147+0000) 2024-02-19T20:37:21.218 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:20.942+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:20.942099+0000) 2024-02-19T20:37:21.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:21.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:21.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:21.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:21.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:21.230+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:21.506 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:21.232+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:21.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:21.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:21.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:21.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:21.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:21.363+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:21.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:21.364+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:21.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:21.752+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.753646+0000) 2024-02-19T20:37:21.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:21.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:21.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:21.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:21.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:21.518+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:21.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:21.519+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:22.138 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:21.944+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:21.945723+0000) 2024-02-19T20:37:22.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:22.167+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:22.168803+0000) 2024-02-19T20:37:22.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:22.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:22.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:22.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:22.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:22.194+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:22.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:22.195+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:22.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:22.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:22.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:22.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:22.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:22.399+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:22.421 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:22.400+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:22.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:22.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:22.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:22.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:22.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:22.515+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:22.742 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:22.516+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:23.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:22.947+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:22.948306+0000) 2024-02-19T20:37:23.011 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:22.755+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:22.756179+0000) 2024-02-19T20:37:23.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:23.169+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.170845+0000) 2024-02-19T20:37:23.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:23.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:23.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:23.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:23.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:23.173+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:23.278 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:23.174+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:23.744 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:23.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:23.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:23.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:23.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:23.516+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:23.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:23.516+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:23.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:23.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:23.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:23.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:23.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:23.412+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:23.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:23.412+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:23.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:23.758+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.759064+0000) 2024-02-19T20:37:24.027 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:23.959+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68375 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:24.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:23.956+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.957374+0000) 2024-02-19T20:37:24.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:23.956+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.957878+0000) 2024-02-19T20:37:24.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:23.950+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.951045+0000) 2024-02-19T20:37:24.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:23.955+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:23.956657+0000) 2024-02-19T20:37:24.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:24.170+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:24.171822+0000) 2024-02-19T20:37:24.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:24.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:24.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:24.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:24.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:24.215+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:24.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:24.216+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:24.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:24.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:24.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:24.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:24.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:24.504+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:24.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:24.505+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:24.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:24.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:24.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:24.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:24.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:24.447+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:24.747 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:24.448+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:25.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:24.759+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:24.760123+0000) 2024-02-19T20:37:25.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:24.950+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:24.951964+0000) 2024-02-19T20:37:25.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:25.172+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:25.173842+0000) 2024-02-19T20:37:25.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:25.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:25.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:25.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:25.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:25.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:25.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:25.184+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:25.748 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:25.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:25.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:25.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:25.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:25.488+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:25.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:25.488+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:25.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:25.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:25.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:25.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:25.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:25.452+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:25.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:25.452+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:26.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:25.953+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:25.954415+0000) 2024-02-19T20:37:26.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:25.761+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:25.762741+0000) 2024-02-19T20:37:26.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:26.175+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.176034+0000) 2024-02-19T20:37:26.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:26.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:26.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:26.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:26.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:26.189+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:26.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:26.190+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:26.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:26.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:26.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:26.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:26.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:26.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:26.751 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:26.491+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:26.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:26.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:26.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:26.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:26.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:26.462+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:26.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:26.462+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:27.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:26.955+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.957005+0000) 2024-02-19T20:37:27.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:26.764+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:26.765691+0000) 2024-02-19T20:37:27.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:27.177+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:27.178128+0000) 2024-02-19T20:37:27.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:27.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:27.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:27.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:27.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:27.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:27.424 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:27.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:27.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:27.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:27.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:27.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:27.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:27.508+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:27.754 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:27.508+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:27.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:27.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:27.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:27.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:27.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:27.472+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:27.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:27.473+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:28.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:27.958+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:27.959006+0000) 2024-02-19T20:37:28.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:27.766+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:27.767869+0000) 2024-02-19T20:37:28.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:28.179+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.180544+0000) 2024-02-19T20:37:28.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:28.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:28.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:28.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:28.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:28.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:28.424 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:28.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:28.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:28.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:28.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:28.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:28.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:28.453+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:28.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:28.454+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:28.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:28.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:28.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:28.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:28.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:28.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:28.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:28.536+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:29.027 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:28.967+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68413 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:29.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:28.964+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.965141+0000) 2024-02-19T20:37:29.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:28.769+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.770456+0000) 2024-02-19T20:37:29.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:28.964+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.965031+0000) 2024-02-19T20:37:29.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:28.963+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:28.964999+0000) 2024-02-19T20:37:29.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:29.180+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:29.181604+0000) 2024-02-19T20:37:29.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:29.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:29.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:29.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:29.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:29.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:29.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:29.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:29.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:29.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:29.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:29.197+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:29.420 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:29.198+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:29.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:29.433+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:29.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:29.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:29.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:29.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:29.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:29.541+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:29.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:29.542+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:30.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:29.770+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:29.771569+0000) 2024-02-19T20:37:30.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:30.182+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:30.183842+0000) 2024-02-19T20:37:30.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:30.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:30.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:30.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:30.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:30.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:30.398 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:30.217+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:30.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:30.218+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:30.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:30.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:30.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:30.411+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:30.760 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:30.411+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:30.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:30.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:30.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:30.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:30.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:30.563+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:30.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:30.564+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:31.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:30.772+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:30.773902+0000) 2024-02-19T20:37:31.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:31.185+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.186028+0000) 2024-02-19T20:37:31.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:31.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:31.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:31.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:31.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:31.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:31.399 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:31.203+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:31.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:31.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:31.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:31.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:31.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:31.542+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:31.763 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:31.543+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:31.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:31.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:31.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:31.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:31.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:31.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:31.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:31.412+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:32.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:31.776+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:31.777547+0000) 2024-02-19T20:37:32.388 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:32.187+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:32.188082+0000) 2024-02-19T20:37:32.388 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:32.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:32.389 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:32.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:32.389 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:32.401+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:32.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:32.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:32.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:32.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:32.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:32.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:32.389 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:32.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:32.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:32.402+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:32.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:32.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:32.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:32.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:32.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:32.545+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:32.766 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:32.545+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:33.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:32.779+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:32.780092+0000) 2024-02-19T20:37:33.353 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:33.189+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.190080+0000) 2024-02-19T20:37:33.354 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:33.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:33.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:33.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:33.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:33.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:33.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:33.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:33.354 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:33.242+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:33.623 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:33.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:33.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:33.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:33.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:33.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:33.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:33.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:33.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:33.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:33.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:33.499+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:33.624 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:33.499+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:33.955 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:33.781+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.782542+0000) 2024-02-19T20:37:34.267 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:33.976+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68454 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:34.267 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:33.973+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.974719+0000) 2024-02-19T20:37:34.268 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:33.972+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.973625+0000) 2024-02-19T20:37:34.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:33.972+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:33.973605+0000) 2024-02-19T20:37:34.268 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:34.190+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:34.191055+0000) 2024-02-19T20:37:34.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:34.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:34.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:34.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:34.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:34.507+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:34.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:34.508+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:34.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:34.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:34.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:34.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:34.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:34.335+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:34.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:34.336+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:34.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:34.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:34.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:34.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:34.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:34.280+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:34.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:34.281+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:34.840 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:34.782+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:34.783861+0000) 2024-02-19T20:37:35.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:35.193+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:35.194506+0000) 2024-02-19T20:37:35.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:35.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:35.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:35.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:35.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:35.300+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:35.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:35.300+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:35.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:35.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:35.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:35.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:35.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:35.260+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:35.432 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:35.261+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:35.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:35.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:35.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:35.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:35.697 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:35.487+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:35.698 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:35.488+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:36.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:35.710+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:35.711978+0000) 2024-02-19T20:37:36.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:35.784+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:35.785916+0000) 2024-02-19T20:37:36.314 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:36.195+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.196924+0000) 2024-02-19T20:37:36.315 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:36.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:36.315 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:36.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:36.315 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:36.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:36.315 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:36.250+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:36.574 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:36.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:36.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:36.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:36.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:36.328+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:36.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:36.328+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:36.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:36.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:36.575 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:36.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:36.576 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:36.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:36.576 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:36.536+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:36.841 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:36.787+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.788538+0000) 2024-02-19T20:37:36.842 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:36.713+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:36.714118+0000) 2024-02-19T20:37:37.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:37.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:37.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:37.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:37.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:37.295+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:37.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:37.296+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:37.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:37.197+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:37.198991+0000) 2024-02-19T20:37:37.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:37.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:37.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:37.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:37.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:37.367+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:37.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:37.367+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:37.701 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:37.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:37.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:37.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:37.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:37.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:37.702 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:37.567+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:38.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:37.715+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:37.716147+0000) 2024-02-19T20:37:38.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:37.789+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:37.790593+0000) 2024-02-19T20:37:38.286 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:38.200+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.201064+0000) 2024-02-19T20:37:38.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:38.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:38.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:38.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:38.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:38.357+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:38.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:38.358+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:38.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:38.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:38.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:38.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:38.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:38.299+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:38.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:38.300+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:38.844 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:38.717+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.718744+0000) 2024-02-19T20:37:38.844 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:38.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:38.844 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:38.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:38.844 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:38.604+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:38.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:38.604+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:38.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:38.791+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.792645+0000) 2024-02-19T20:37:39.140 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:38.983+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68492 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:39.140 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:38.979+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.980612+0000) 2024-02-19T20:37:39.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:38.980+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.981208+0000) 2024-02-19T20:37:39.141 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:38.980+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:38.981772+0000) 2024-02-19T20:37:39.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:39.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:39.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:39.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:39.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:39.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:39.434 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:39.332+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:39.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:39.201+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:39.202060+0000) 2024-02-19T20:37:39.435 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:39.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:39.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:39.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:39.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:39.336+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:39.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:39.336+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:39.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:39.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:39.706 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:39.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:39.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:39.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:39.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:39.555+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:40.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:39.719+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:39.720841+0000) 2024-02-19T20:37:40.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:39.792+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:39.793703+0000) 2024-02-19T20:37:40.328 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:40.203+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:40.204037+0000) 2024-02-19T20:37:40.328 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:40.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:40.328 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:40.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:40.329 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:40.341+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:40.583 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:40.342+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:40.583 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:40.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:40.583 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:40.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:40.583 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:40.369+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:40.583 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:40.370+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:40.841 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:40.723+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:40.724529+0000) 2024-02-19T20:37:40.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:40.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:40.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:40.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:40.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:40.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:40.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:40.597+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:40.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:40.794+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:40.795765+0000) 2024-02-19T20:37:41.438 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:41.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:41.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:41.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:41.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:41.331+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:41.439 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:41.331+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:41.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:41.205+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.206512+0000) 2024-02-19T20:37:41.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:41.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:41.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:41.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:41.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:41.377+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:41.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:41.378+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:41.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:41.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:41.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:41.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:41.712 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:41.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:41.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:41.631+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:42.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:41.796+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.797765+0000) 2024-02-19T20:37:42.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:41.725+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:41.726760+0000) 2024-02-19T20:37:42.318 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:42.208+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:42.209126+0000) 2024-02-19T20:37:42.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:42.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:42.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:42.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:42.576 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:42.332+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:42.576 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:42.333+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:42.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:42.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:42.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:42.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:42.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:42.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:42.577 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:42.362+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:42.841 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:42.729+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:42.730377+0000) 2024-02-19T20:37:42.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:42.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:42.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:42.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:42.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:42.610+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:42.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:42.611+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:42.842 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:42.799+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:42.800338+0000) 2024-02-19T20:37:43.438 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:43.211+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.212803+0000) 2024-02-19T20:37:43.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:43.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:43.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:43.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:43.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:43.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:43.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:43.355+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:43.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:43.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:43.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:43.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:43.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:43.347+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:43.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:43.347+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:43.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:43.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:43.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:43.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:43.719 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:43.643+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:43.720 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:43.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:43.971 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:43.802+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.803944+0000) 2024-02-19T20:37:43.972 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:43.732+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.734008+0000) 2024-02-19T20:37:44.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:43.992+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68533 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:44.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:43.988+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.989503+0000) 2024-02-19T20:37:44.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:43.988+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.989919+0000) 2024-02-19T20:37:44.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:43.988+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:43.989375+0000) 2024-02-19T20:37:44.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:44.212+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:44.213783+0000) 2024-02-19T20:37:44.624 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:44.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:44.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:44.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:44.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:44.385+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:44.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:44.386+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:44.625 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:44.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:44.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:44.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:44.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:44.359+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:44.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:44.360+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:45.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:44.734+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:44.735064+0000) 2024-02-19T20:37:45.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:44.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:45.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:44.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:45.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:44.682+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:45.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:44.683+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:45.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:44.803+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:44.804979+0000) 2024-02-19T20:37:45.342 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:45.215+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:45.216301+0000) 2024-02-19T20:37:45.342 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:45.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:45.342 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:45.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:45.342 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:45.355+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:45.624 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:45.356+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:45.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:45.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:45.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:45.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:45.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:45.426+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:45.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:45.427+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:46.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:45.737+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:45.738451+0000) 2024-02-19T20:37:46.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:45.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:46.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:45.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:46.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:45.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:46.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:45.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:46.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:45.806+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:45.807088+0000) 2024-02-19T20:37:46.316 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:46.217+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.218235+0000) 2024-02-19T20:37:46.316 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:46.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:46.316 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:46.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:46.316 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:46.329+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:46.575 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:46.330+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:46.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:46.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:46.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:46.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:46.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:46.452+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:46.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:46.453+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:46.842 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:46.741+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.742063+0000) 2024-02-19T20:37:46.843 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:46.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:46.843 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:46.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:46.843 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:46.700+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:46.843 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:46.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:46.844 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:46.808+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:46.809118+0000) 2024-02-19T20:37:47.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:47.219+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:47.220230+0000) 2024-02-19T20:37:47.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:47.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:47.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:47.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:47.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:47.298+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:47.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:47.298+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:47.729 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:47.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:47.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:47.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:47.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:47.653+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:47.730 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:47.653+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:47.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:47.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:47.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:47.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:47.730 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:47.486+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:47.731 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:47.487+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:48.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:47.743+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:47.744094+0000) 2024-02-19T20:37:48.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:47.810+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:47.811089+0000) 2024-02-19T20:37:48.323 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:48.221+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.222284+0000) 2024-02-19T20:37:48.324 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:48.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:48.324 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:48.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:48.324 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:48.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:48.576 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:48.338+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:48.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:48.496+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:48.576 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:48.496+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:48.577 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:48.496+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:48.577 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:48.497+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:48.842 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:48.746+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.747573+0000) 2024-02-19T20:37:48.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:48.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:48.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:48.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:48.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:48.629+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:48.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:48.629+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:48.845 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:48.812+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.813172+0000) 2024-02-19T20:37:49.104 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:49.000+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68571 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:49.105 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:48.996+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.997764+0000) 2024-02-19T20:37:49.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:48.999+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:49.000055+0000) 2024-02-19T20:37:49.106 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:48.996+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:48.997911+0000) 2024-02-19T20:37:49.364 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:49.222+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:49.223271+0000) 2024-02-19T20:37:49.364 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:49.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:49.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:49.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:49.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:49.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:49.625 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:49.589+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:49.626 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:49.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:49.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:49.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:49.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:49.378+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:49.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:49.378+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:49.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:49.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:49.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:49.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:49.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:49.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:49.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:49.525+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:50.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:49.748+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:49.749202+0000) 2024-02-19T20:37:50.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:49.813+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:49.814169+0000) 2024-02-19T20:37:50.380 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:50.225+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:50.226698+0000) 2024-02-19T20:37:50.380 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:50.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:50.380 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:50.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:50.380 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:50.393+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:50.658 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:50.394+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:50.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:50.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:50.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:50.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:50.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:50.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:50.659 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:50.608+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:50.659 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:50.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:50.660 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:50.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:50.660 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:50.507+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:50.660 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:50.508+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:51.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:50.750+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:50.751250+0000) 2024-02-19T20:37:51.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:50.815+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:50.816173+0000) 2024-02-19T20:37:51.355 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:51.227+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.228753+0000) 2024-02-19T20:37:51.625 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:51.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:51.625 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:51.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:51.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:51.368+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:51.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:51.369+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:51.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:51.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:51.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:51.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:51.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:51.504+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:51.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:51.505+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:52.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:51.752+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.753300+0000) 2024-02-19T20:37:52.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:51.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:52.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:51.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:52.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:51.641+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:52.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:51.642+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:52.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:51.818+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:51.819791+0000) 2024-02-19T20:37:52.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:52.231+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:52.232297+0000) 2024-02-19T20:37:52.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:52.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:52.366 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:52.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:52.367 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:52.379+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:52.625 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:52.380+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:52.625 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:52.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:52.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:52.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:52.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:52.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:52.626 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:52.481+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:53.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:52.755+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:52.756683+0000) 2024-02-19T20:37:53.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:52.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:53.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:52.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:53.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:52.658+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:53.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:52.658+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:53.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:52.821+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:52.822033+0000) 2024-02-19T20:37:53.345 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:53.234+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:53.235776+0000) 2024-02-19T20:37:53.345 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:53.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:53.345 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:53.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:53.346 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:53.358+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:53.602 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:53.359+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:53.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:53.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:53.602 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:53.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:53.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:53.511+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:53.603 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:53.513+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:53.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:53.759+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:53.760196+0000) 2024-02-19T20:37:53.988 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:53.615+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:53.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:53.615+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:53.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:53.615+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:53.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:53.616+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:53.989 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:53.823+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:53.824065+0000) 2024-02-19T20:37:54.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:54.008+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68612 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:54.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:54.005+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.006349+0000) 2024-02-19T20:37:54.278 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:54.005+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.006051+0000) 2024-02-19T20:37:54.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:54.004+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.005552+0000) 2024-02-19T20:37:54.278 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:54.236+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.237454+0000) 2024-02-19T20:37:54.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:54.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:54.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:54.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:54.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:54.372+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:54.626 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:54.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:54.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:54.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:54.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:54.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:54.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:54.482+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:54.627 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:54.482+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:54.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:54.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:54.627 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:54.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:54.628 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:54.608+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:54.628 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:54.609+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:55.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:54.760+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.761855+0000) 2024-02-19T20:37:55.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:54.824+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:54.825802+0000) 2024-02-19T20:37:55.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:55.238+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:55.239432+0000) 2024-02-19T20:37:55.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:55.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:55.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:55.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:55.390 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:55.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:55.749 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:55.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:55.749 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:55.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:55.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:55.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:55.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:55.624+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:55.750 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:55.624+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:55.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:55.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:55.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:55.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:55.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:55.480+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:55.750 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:55.481+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:55.751 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:55.762+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:55.763877+0000) 2024-02-19T20:37:56.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:55.826+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:55.827805+0000) 2024-02-19T20:37:56.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:56.240+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.241492+0000) 2024-02-19T20:37:56.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:56.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:56.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:56.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:56.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:56.448+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:56.752 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:56.448+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:56.752 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:56.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:56.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:56.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:56.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:56.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:56.753 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:56.606+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:56.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:56.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:56.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:56.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:56.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:56.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:56.753 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:56.484+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:56.754 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:56.765+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.766367+0000) 2024-02-19T20:37:57.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:56.828+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:56.829815+0000) 2024-02-19T20:37:57.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:57.242+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:57.243929+0000) 2024-02-19T20:37:57.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:57.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:57.754 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:57.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:57.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:57.444+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:57.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:57.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:57.755 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:57.596+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:57.756 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:57.597+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:57.756 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:57.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:57.756 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:57.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:57.756 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:57.481+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:57.756 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:57.482+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:58.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:57.830+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:57.831879+0000) 2024-02-19T20:37:58.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:57.767+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:57.768943+0000) 2024-02-19T20:37:58.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:58.246+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:58.247615+0000) 2024-02-19T20:37:58.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:58.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:58.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:58.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:58.757 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:58.490+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:58.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:58.491+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:58.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:58.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:58.758 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:58.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:58.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:58.432+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:58.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:58.433+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:37:58.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:58.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:58.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:58.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:58.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:58.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:58.760 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:58.578+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:59.008 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:37:59.016+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68650 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:37:59.009 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:59.013+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.014923+0000) 2024-02-19T20:37:59.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:58.832+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:58.833990+0000) 2024-02-19T20:37:59.010 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:59.012+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.013679+0000) 2024-02-19T20:37:59.010 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:58.770+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:58.771109+0000) 2024-02-19T20:37:59.011 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:59.013+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.014894+0000) 2024-02-19T20:37:59.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:59.248+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.249063+0000) 2024-02-19T20:37:59.757 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:59.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:59.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:59.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:59.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:59.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:37:59.758 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:59.574+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:59.758 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:59.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:59.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:59.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:59.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:59.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:37:59.759 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:37:59.458+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:37:59.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:59.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:59.759 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:59.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:59.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:59.456+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:37:59.760 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:59.457+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:00.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:37:59.771+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.772252+0000) 2024-02-19T20:38:00.028 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:37:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:37:59.834+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:37:59.835444+0000) 2024-02-19T20:38:00.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:00.250+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:00.251065+0000) 2024-02-19T20:38:00.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:00.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:00.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:00.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:00.761 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:00.457+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:00.762 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:00.458+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:00.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:00.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:00.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:00.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:00.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:00.555+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:00.762 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:00.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:00.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:00.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:00.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:00.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:00.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:00.507+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:00.763 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:00.507+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:01.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:00.774+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:00.775763+0000) 2024-02-19T20:38:01.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:00.836+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:00.837475+0000) 2024-02-19T20:38:01.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:01.253+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.254567+0000) 2024-02-19T20:38:01.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:01.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:01.763 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:01.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:01.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:01.474+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:01.764 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:01.475+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:01.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:01.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:01.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:01.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:01.764 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:01.520+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:01.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:01.521+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:01.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:01.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:01.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:01.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:01.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:01.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:01.765 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:01.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:02.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:01.839+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.840927+0000) 2024-02-19T20:38:02.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:01.777+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:01.778038+0000) 2024-02-19T20:38:02.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:02.255+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:02.256888+0000) 2024-02-19T20:38:02.765 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:02.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:02.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:02.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:02.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:02.502+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:02.766 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:02.502+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:02.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:02.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:02.766 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:02.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:02.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:02.483+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:02.767 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:02.484+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:02.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:02.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:02.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:02.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:02.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:02.489+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:02.767 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:02.490+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:03.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:02.779+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:02.780270+0000) 2024-02-19T20:38:03.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:02.843+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:02.844453+0000) 2024-02-19T20:38:03.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:03.258+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:03.259160+0000) 2024-02-19T20:38:03.440 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:03.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:03.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:03.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:03.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:03.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:03.768 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:03.454+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:03.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:03.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:03.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:03.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:03.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:03.537+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:03.769 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:03.537+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:03.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:03.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:03.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:03.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:03.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:03.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:03.770 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:03.527+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:04.027 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:04.024+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68691 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:04.028 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:04.020+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.021473+0000) 2024-02-19T20:38:04.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:03.782+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:03.783440+0000) 2024-02-19T20:38:04.029 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:04.021+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.022481+0000) 2024-02-19T20:38:04.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:03.846+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:03.847915+0000) 2024-02-19T20:38:04.029 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:04.020+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.021373+0000) 2024-02-19T20:38:04.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:04.259+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.260228+0000) 2024-02-19T20:38:04.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:04.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:04.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:04.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:04.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:04.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:04.770 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:04.419+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:04.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:04.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:04.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:04.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:04.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:04.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:04.771 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:04.544+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:04.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:04.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:04.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:04.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:04.772 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:04.546+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:04.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:04.546+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:05.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:04.783+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.784734+0000) 2024-02-19T20:38:05.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:04.848+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:04.849631+0000) 2024-02-19T20:38:05.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:05.261+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:05.262416+0000) 2024-02-19T20:38:05.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:05.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:05.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:05.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:05.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:05.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:05.772 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:05.445+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:05.772 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:05.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:05.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:05.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:05.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:05.521+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:05.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:05.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:05.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:05.524+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:05.773 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:05.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:05.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:05.525+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:05.774 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:05.525+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:06.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:05.786+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:05.787037+0000) 2024-02-19T20:38:06.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:05.851+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:05.852234+0000) 2024-02-19T20:38:06.460 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:06.263+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.264921+0000) 2024-02-19T20:38:06.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:06.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:06.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:06.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:06.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:06.474+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:06.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:06.475+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:06.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:06.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:06.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:06.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:06.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:06.571+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:06.776 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:06.571+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:06.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:06.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:06.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:06.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:06.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:06.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:06.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:06.543+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:07.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:06.854+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.854245+0000) 2024-02-19T20:38:07.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:06.789+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:06.789500+0000) 2024-02-19T20:38:07.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:07.267+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:07.267045+0000) 2024-02-19T20:38:07.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:07.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:07.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:07.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:07.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:07.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:07.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:07.589+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:07.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:07.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:07.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:07.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:07.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:07.617+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:07.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:07.618+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:07.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:07.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:07.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:07.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:07.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:07.455+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:07.779 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:07.456+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:08.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:07.855+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:07.856879+0000) 2024-02-19T20:38:08.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:07.791+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:07.792175+0000) 2024-02-19T20:38:08.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:08.268+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:08.269494+0000) 2024-02-19T20:38:08.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:08.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:08.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:08.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:08.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:08.644+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:08.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:08.645+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:08.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:08.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:08.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:08.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:08.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:08.459+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:08.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:08.460+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:08.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:08.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:08.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:08.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:08.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:08.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:08.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:08.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:09.036 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:09.032+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68729 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:09.036 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:09.029+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.030027+0000) 2024-02-19T20:38:09.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:08.793+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:08.794762+0000) 2024-02-19T20:38:09.037 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:09.029+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.029987+0000) 2024-02-19T20:38:09.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:08.857+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:08.859004+0000) 2024-02-19T20:38:09.038 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:09.028+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.029542+0000) 2024-02-19T20:38:09.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:09.269+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.270731+0000) 2024-02-19T20:38:09.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:09.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:09.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:09.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:09.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:09.468+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:09.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:09.469+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:09.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:09.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:09.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:09.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:09.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:09.564+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:09.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:09.565+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:09.778 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:09.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:09.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:09.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:09.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:09.671+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:09.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:09.671+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:10.039 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:09.794+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.795910+0000) 2024-02-19T20:38:10.039 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:09.859+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:09.860771+0000) 2024-02-19T20:38:10.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:10.272+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:10.273284+0000) 2024-02-19T20:38:10.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:10.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:10.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:10.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:10.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:10.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:10.680 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:10.440+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:10.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:10.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:10.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:10.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:10.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:10.562+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:10.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:10.563+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:11.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:10.861+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:10.862957+0000) 2024-02-19T20:38:11.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:10.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:11.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:10.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:11.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:10.694+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:11.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:10.694+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:11.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:10.796+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:10.797971+0000) 2024-02-19T20:38:11.427 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:11.274+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.275320+0000) 2024-02-19T20:38:11.427 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:11.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:11.427 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:11.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:11.428 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:11.441+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:11.713 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:11.441+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:11.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:11.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:11.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:11.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:11.713 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:11.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:11.714 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:11.607+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:12.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:11.864+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.865423+0000) 2024-02-19T20:38:12.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:11.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:12.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:11.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:12.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:11.726+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:12.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:11.727+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:12.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:11.798+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:11.800003+0000) 2024-02-19T20:38:12.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:12.276+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:12.277445+0000) 2024-02-19T20:38:12.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:12.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:12.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:12.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:12.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:12.416+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:12.739 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:12.417+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:12.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:12.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:12.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:12.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:12.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:12.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:12.740 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:12.578+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:13.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:12.866+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:12.867403+0000) 2024-02-19T20:38:13.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:12.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:13.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:12.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:13.027 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:12.752+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:13.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:12.753+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:13.028 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:12.801+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:12.802066+0000) 2024-02-19T20:38:13.412 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:13.278+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:13.279504+0000) 2024-02-19T20:38:13.773 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:13.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:13.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:13.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:13.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:13.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:13.774 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:13.573+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:13.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:13.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:13.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:13.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:13.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:13.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:13.774 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:13.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:14.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:13.868+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:13.869440+0000) 2024-02-19T20:38:14.024 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:14.037+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.038034+0000) 2024-02-19T20:38:14.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:13.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:14.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:13.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:14.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:13.787+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:14.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:13.788+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:14.025 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:13.803+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:13.804083+0000) 2024-02-19T20:38:14.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:14.037+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.038657+0000) 2024-02-19T20:38:14.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:14.279+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.280534+0000) 2024-02-19T20:38:14.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:14.037+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.038504+0000) 2024-02-19T20:38:14.278 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:14.041+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68770 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:14.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:14.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:14.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:14.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:14.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:14.477+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:14.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:14.478+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:14.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:14.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:14.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:14.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:14.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:14.560+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:14.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:14.561+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:15.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:14.869+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.870626+0000) 2024-02-19T20:38:15.144 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:14.804+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:14.805542+0000) 2024-02-19T20:38:15.144 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:14.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:15.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:14.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:15.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:14.816+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:15.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:14.817+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:15.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:15.281+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:15.282534+0000) 2024-02-19T20:38:15.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:15.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:15.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:15.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:15.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:15.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:15.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:15.460+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:15.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:15.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:15.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:15.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:15.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:15.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:15.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:15.516+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:16.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:15.871+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:15.872633+0000) 2024-02-19T20:38:16.144 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:15.807+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:15.808081+0000) 2024-02-19T20:38:16.144 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:15.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:16.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:15.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:16.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:15.833+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:16.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:15.834+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:16.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:16.284+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.285098+0000) 2024-02-19T20:38:16.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:16.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:16.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:16.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:16.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:16.454+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:16.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:16.455+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:16.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:16.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:16.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:16.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:16.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:16.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:16.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:16.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:17.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:16.874+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.875208+0000) 2024-02-19T20:38:17.144 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:16.809+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:16.810910+0000) 2024-02-19T20:38:17.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:16.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:17.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:16.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:17.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:16.881+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:17.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:16.882+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:17.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:17.286+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:17.287182+0000) 2024-02-19T20:38:17.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:17.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:17.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:17.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:17.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:17.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:17.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:17.500+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:17.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:17.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:17.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:17.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:17.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:17.559+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:17.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:17.559+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:18.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:17.876+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:17.877704+0000) 2024-02-19T20:38:18.144 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:17.812+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:17.813442+0000) 2024-02-19T20:38:18.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:17.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:18.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:17.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:18.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:17.885+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:18.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:17.886+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:18.474 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:18.288+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:18.289287+0000) 2024-02-19T20:38:18.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:18.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:18.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:18.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:18.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:18.499+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:18.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:18.500+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:18.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:18.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:18.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:18.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:18.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:18.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:18.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:18.575+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:19.054 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:19.050+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68808 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:19.055 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:19.045+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.046638+0000) 2024-02-19T20:38:19.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:18.879+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:18.880157+0000) 2024-02-19T20:38:19.055 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:19.045+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.046925+0000) 2024-02-19T20:38:19.055 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:18.814+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:18.815961+0000) 2024-02-19T20:38:19.056 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:18.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:19.056 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:18.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:19.056 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:18.898+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:19.056 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:18.899+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:19.056 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:19.046+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.047027+0000) 2024-02-19T20:38:19.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:19.289+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.290547+0000) 2024-02-19T20:38:19.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:19.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:19.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:19.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:19.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:19.473+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:19.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:19.474+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:19.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:19.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:19.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:19.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:19.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:19.543+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:19.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:19.544+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:20.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:19.880+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.881386+0000) 2024-02-19T20:38:20.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:19.816+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:19.817349+0000) 2024-02-19T20:38:20.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:19.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:20.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:19.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:20.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:19.906+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:20.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:19.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:20.429 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:20.293+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:20.294092+0000) 2024-02-19T20:38:20.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:20.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:20.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:20.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:20.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:20.443+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:20.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:20.443+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:20.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:20.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:20.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:20.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:20.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:20.574+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:20.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:20.574+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:21.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:20.883+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:20.884796+0000) 2024-02-19T20:38:21.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:20.819+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:20.820708+0000) 2024-02-19T20:38:21.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:20.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:21.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:20.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:21.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:20.860+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:21.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:20.861+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:21.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:21.295+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.296084+0000) 2024-02-19T20:38:21.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:21.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:21.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:21.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:21.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:21.426+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:21.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:21.427+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:21.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:21.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:21.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:21.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:21.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:21.570+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:21.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:21.570+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:22.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:21.887+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.888231+0000) 2024-02-19T20:38:22.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:21.823+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:21.824179+0000) 2024-02-19T20:38:22.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:21.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:22.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:21.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:22.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:21.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:22.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:21.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:22.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:22.297+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:22.298247+0000) 2024-02-19T20:38:22.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:22.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:22.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:22.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:22.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:22.461+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:22.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:22.462+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:22.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:22.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:22.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:22.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:22.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:22.607+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:22.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:22.607+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:23.144 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:22.890+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:22.891539+0000) 2024-02-19T20:38:23.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:22.826+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:22.827685+0000) 2024-02-19T20:38:23.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:22.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:23.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:22.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:23.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:22.827+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:23.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:22.828+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:23.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:23.300+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:23.301832+0000) 2024-02-19T20:38:23.420 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:23.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:23.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:23.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:23.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:23.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:23.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:23.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:23.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:23.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:23.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:23.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:23.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:23.640+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:23.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:23.640+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:24.064 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:24.058+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68850 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:24.064 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:24.054+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.055784+0000) 2024-02-19T20:38:24.065 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:23.893+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:23.894925+0000) 2024-02-19T20:38:24.065 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:24.055+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.056557+0000) 2024-02-19T20:38:24.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:23.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:24.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:23.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:24.065 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:23.825+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:24.066 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:23.826+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:24.066 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:23.830+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:23.831107+0000) 2024-02-19T20:38:24.066 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:24.054+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.055335+0000) 2024-02-19T20:38:24.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:24.302+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.303081+0000) 2024-02-19T20:38:24.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:24.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:24.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:24.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:24.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:24.439+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:24.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:24.440+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:24.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:24.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:24.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:24.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:24.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:24.597+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:24.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:24.597+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:25.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:24.895+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.896735+0000) 2024-02-19T20:38:25.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:24.831+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:24.832843+0000) 2024-02-19T20:38:25.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:24.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:25.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:24.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:25.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:24.844+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:25.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:24.845+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:25.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:25.304+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:25.305229+0000) 2024-02-19T20:38:25.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:25.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:25.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:25.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:25.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:25.432+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:25.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:25.433+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:25.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:25.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:25.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:25.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:25.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:25.567+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:25.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:25.567+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:26.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:25.897+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:25.898822+0000) 2024-02-19T20:38:26.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:25.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:26.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:25.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:26.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:25.807+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:26.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:25.808+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:26.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:25.835+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:25.836408+0000) 2024-02-19T20:38:26.446 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:26.306+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.307207+0000) 2024-02-19T20:38:26.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:26.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:26.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:26.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:26.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:26.582+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:26.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:26.583+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:26.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:26.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:26.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:26.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:26.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:26.460+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:26.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:26.460+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:27.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:26.899+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.900907+0000) 2024-02-19T20:38:27.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:26.837+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:26.838413+0000) 2024-02-19T20:38:27.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:26.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:27.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:26.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:27.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:26.849+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:27.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:26.850+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:27.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:27.309+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:27.310639+0000) 2024-02-19T20:38:27.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:27.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:27.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:27.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:27.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:27.498+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:27.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:27.499+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:27.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:27.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:27.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:27.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:27.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:27.540+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:27.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:27.540+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:28.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:27.903+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:27.904582+0000) 2024-02-19T20:38:28.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:27.840+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:27.842005+0000) 2024-02-19T20:38:28.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:27.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:28.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:27.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:28.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:27.846+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:28.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:27.847+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:28.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:28.311+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:28.312928+0000) 2024-02-19T20:38:28.444 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:28.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:28.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:28.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:28.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:28.458+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:28.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:28.459+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:28.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:28.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:28.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:28.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:28.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:28.538+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:28.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:28.539+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:29.145 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:29.066+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68888 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:29.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:29.063+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.064438+0000) 2024-02-19T20:38:29.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:28.905+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:28.906745+0000) 2024-02-19T20:38:29.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:29.063+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.064607+0000) 2024-02-19T20:38:29.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:28.843+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:28.844161+0000) 2024-02-19T20:38:29.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:28.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:29.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:28.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:29.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:28.882+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:29.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:28.883+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:29.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:29.062+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.063762+0000) 2024-02-19T20:38:29.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:29.313+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.314183+0000) 2024-02-19T20:38:29.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:29.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:29.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:29.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:29.398 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:29.412+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:29.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:29.412+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:29.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:29.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:29.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:29.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:29.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:29.516+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:29.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:29.517+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:30.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:29.907+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.908030+0000) 2024-02-19T20:38:30.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:29.844+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:29.845927+0000) 2024-02-19T20:38:30.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:29.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:30.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:29.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:30.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:29.870+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:30.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:29.871+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:30.409 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:30.315+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:30.316359+0000) 2024-02-19T20:38:30.410 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:30.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:30.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:30.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:30.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:30.423+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:30.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:30.424+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:30.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:30.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:30.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:30.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:30.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:30.506+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:30.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:30.507+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:31.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:30.909+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:30.910014+0000) 2024-02-19T20:38:31.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:30.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:31.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:30.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:31.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:30.840+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:31.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:30.841+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:31.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:30.847+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:30.848153+0000) 2024-02-19T20:38:31.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:31.317+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.318435+0000) 2024-02-19T20:38:31.492 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:31.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:31.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:31.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:31.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:31.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:31.493 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:31.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:31.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:31.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:31.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:31.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:31.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:31.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:31.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:31.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:32.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:31.911+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.912101+0000) 2024-02-19T20:38:32.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:31.849+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:31.850374+0000) 2024-02-19T20:38:32.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:31.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:32.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:31.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:32.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:31.859+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:32.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:31.859+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:32.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:32.321+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:32.322111+0000) 2024-02-19T20:38:32.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:32.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:32.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:32.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:32.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:32.433+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:32.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:32.434+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:32.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:32.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:32.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:32.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:32.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:32.509+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:32.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:32.509+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:33.145 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:32.913+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:32.914691+0000) 2024-02-19T20:38:33.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:32.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:33.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:32.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:33.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:32.826+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:33.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:32.827+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:33.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:32.852+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:32.853985+0000) 2024-02-19T20:38:33.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:33.323+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:33.324421+0000) 2024-02-19T20:38:33.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:33.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:33.453 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:33.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:33.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:33.466+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:33.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:33.467+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:33.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:33.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:33.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:33.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:33.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:33.552+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:33.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:33.552+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:34.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:34.075+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68929 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:34.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:34.071+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.072323+0000) 2024-02-19T20:38:34.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:33.916+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:33.917282+0000) 2024-02-19T20:38:34.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:34.072+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.073211+0000) 2024-02-19T20:38:34.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:33.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:34.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:33.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:34.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:33.852+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:34.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:33.853+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:34.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:33.855+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:33.856441+0000) 2024-02-19T20:38:34.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:34.071+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.072071+0000) 2024-02-19T20:38:34.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:34.325+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.326384+0000) 2024-02-19T20:38:34.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:34.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:34.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:34.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:34.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:34.418+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:34.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:34.419+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:34.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:34.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:34.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:34.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:34.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:34.561+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:34.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:34.562+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:35.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:34.917+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.918525+0000) 2024-02-19T20:38:35.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:34.856+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:34.857772+0000) 2024-02-19T20:38:35.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:34.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:35.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:34.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:35.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:34.876+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:35.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:34.877+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:35.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:35.327+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:35.328981+0000) 2024-02-19T20:38:35.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:35.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:35.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:35.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:35.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:35.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:35.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:35.546+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:35.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:35.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:35.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:35.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:35.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:35.469+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:35.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:35.470+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:36.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:35.921+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:35.922083+0000) 2024-02-19T20:38:36.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:35.859+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:35.860255+0000) 2024-02-19T20:38:36.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:35.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:36.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:35.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:36.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:35.905+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:36.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:35.906+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:36.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:36.329+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.331007+0000) 2024-02-19T20:38:36.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:36.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:36.418 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:36.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:36.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:36.431+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:36.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:36.432+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:36.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:36.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:36.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:36.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:36.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:36.566+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:36.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:36.566+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:37.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:36.924+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.925685+0000) 2024-02-19T20:38:37.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:36.861+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:36.862871+0000) 2024-02-19T20:38:37.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:36.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:37.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:36.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:37.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:36.945+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:37.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:36.945+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:37.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:37.333+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:37.334616+0000) 2024-02-19T20:38:37.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:37.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:37.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:37.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:37.407 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:37.420+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:37.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:37.421+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:37.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:37.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:37.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:37.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:37.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:37.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:37.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:37.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:38.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:37.926+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:37.927742+0000) 2024-02-19T20:38:38.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:37.863+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:37.864877+0000) 2024-02-19T20:38:38.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:37.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:38.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:37.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:38.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:37.910+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:38.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:37.911+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:38.496 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:38.335+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:38.336797+0000) 2024-02-19T20:38:38.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:38.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:38.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:38.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:38.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:38.403+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:38.497 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:38.404+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:38.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:38.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:38.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:38.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:38.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:38.510+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:38.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:38.511+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:39.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:39.079+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.080349+0000) 2024-02-19T20:38:39.147 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:39.084+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 68967 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:39.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:38.929+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:38.930236+0000) 2024-02-19T20:38:39.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:39.079+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.080751+0000) 2024-02-19T20:38:39.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:38.865+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:38.866904+0000) 2024-02-19T20:38:39.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:38.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:39.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:38.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:39.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:38.958+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:39.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:38.959+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:39.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:39.079+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.080465+0000) 2024-02-19T20:38:39.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:39.337+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.338069+0000) 2024-02-19T20:38:39.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:39.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:39.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:39.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:39.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:39.376+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:39.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:39.377+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:39.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:39.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:39.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:39.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:39.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:39.533+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:39.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:39.533+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:40.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:39.930+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.931640+0000) 2024-02-19T20:38:40.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:39.867+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:39.868692+0000) 2024-02-19T20:38:40.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:40.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:40.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:40.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:40.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:40.008+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:40.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:40.009+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:40.502 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:40.339+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:40.340038+0000) 2024-02-19T20:38:40.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:40.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:40.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:40.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:40.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:40.399+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:40.503 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:40.400+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:40.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:40.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:40.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:40.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:40.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:40.546+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:40.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:40.547+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:41.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:40.933+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:40.934133+0000) 2024-02-19T20:38:41.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:40.870+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:40.871184+0000) 2024-02-19T20:38:41.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:41.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:41.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:41.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:41.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:41.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:41.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:41.043+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:41.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:41.341+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.342463+0000) 2024-02-19T20:38:41.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:41.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:41.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:41.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:41.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:41.363+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:41.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:41.364+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:41.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:41.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:41.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:41.577+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:41.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:41.578+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:41.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:41.578+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:42.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:41.935+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.936199+0000) 2024-02-19T20:38:42.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:41.872+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:41.873292+0000) 2024-02-19T20:38:42.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:42.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:42.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:42.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:42.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:42.093+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:42.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:42.094+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:42.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:42.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:42.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:42.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:42.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:42.320+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:42.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:42.321+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:42.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:42.344+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:42.345051+0000) 2024-02-19T20:38:42.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:42.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:42.779 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:42.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:42.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:42.583+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:42.780 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:42.583+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:43.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:42.937+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:42.938331+0000) 2024-02-19T20:38:43.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:42.874+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:42.875452+0000) 2024-02-19T20:38:43.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:43.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:43.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:43.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:43.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:43.101+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:43.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:43.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:43.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:43.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:43.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:43.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:43.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:43.277+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:43.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:43.278+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:43.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:43.346+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:43.347728+0000) 2024-02-19T20:38:43.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:43.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:43.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:43.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:43.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:43.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:43.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:43.589+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:44.146 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:44.092+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69008 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:44.147 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:44.088+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.089570+0000) 2024-02-19T20:38:44.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:43.940+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:43.941977+0000) 2024-02-19T20:38:44.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:44.089+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.090269+0000) 2024-02-19T20:38:44.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:43.877+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:43.878157+0000) 2024-02-19T20:38:44.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:44.088+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.089423+0000) 2024-02-19T20:38:44.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:44.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:44.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:44.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:44.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:44.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:44.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:44.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:44.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:44.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:44.505 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:44.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:44.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:44.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:44.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:44.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:44.506 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:44.347+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.348913+0000) 2024-02-19T20:38:44.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:44.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:44.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:44.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:44.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:44.556+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:44.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:44.556+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:45.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:44.942+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.943254+0000) 2024-02-19T20:38:45.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:44.878+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:44.879788+0000) 2024-02-19T20:38:45.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:45.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:45.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:45.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:45.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:45.094+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:45.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:45.094+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:45.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:45.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:45.508 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:45.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:45.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:45.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:45.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:45.264+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:45.509 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:45.350+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:45.351343+0000) 2024-02-19T20:38:45.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:45.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:45.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:45.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:45.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:45.599+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:45.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:45.600+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:46.124 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:45.944+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:45.945277+0000) 2024-02-19T20:38:46.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:45.880+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:45.881867+0000) 2024-02-19T20:38:46.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:46.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:46.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:46.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:46.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:46.138+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:46.510 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:46.139+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:46.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:46.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:46.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:46.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:46.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:46.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:46.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:46.290+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:46.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:46.352+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.353394+0000) 2024-02-19T20:38:46.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:46.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:46.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:46.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:46.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:46.575+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:46.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:46.576+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:47.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:46.882+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.883909+0000) 2024-02-19T20:38:47.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:46.946+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:46.947314+0000) 2024-02-19T20:38:47.510 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:47.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:47.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:47.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:47.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:47.274+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:18.703822+0000 (oldest deadline 2024-02-19T20:28:38.703822+0000) 2024-02-19T20:38:47.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:47.275+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:47.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:47.356+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:47.357058+0000) 2024-02-19T20:38:47.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:47.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:47.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:47.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:47.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:47.163+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:47.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:47.164+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:47.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:47.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:47.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:47.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:47.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:47.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:47.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:47.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:48.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:47.948+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:47.949357+0000) 2024-02-19T20:38:48.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:47.884+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:47.885946+0000) 2024-02-19T20:38:48.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:48.319+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:48.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:48.359+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:48.360721+0000) 2024-02-19T20:38:48.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:48.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:48.511 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:48.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:48.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:48.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:48.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:48.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:48.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:48.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:48.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:48.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:48.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:48.614+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:48.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:48.615+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:49.103 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:49.100+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69048 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:49.103 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:49.096+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.097408+0000) 2024-02-19T20:38:49.103 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:48.887+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:48.888101+0000) 2024-02-19T20:38:49.104 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:49.095+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.096953+0000) 2024-02-19T20:38:49.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:48.952+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:48.953012+0000) 2024-02-19T20:38:49.104 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:49.096+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.097969+0000) 2024-02-19T20:38:49.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:49.353+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:49.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:49.361+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.362642+0000) 2024-02-19T20:38:49.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:49.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:49.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:49.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:49.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:49.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:49.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:49.220+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:49.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:49.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:49.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:49.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:49.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:49.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:49.778 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:49.637+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:50.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:49.888+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.889187+0000) 2024-02-19T20:38:50.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:49.953+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:49.954925+0000) 2024-02-19T20:38:50.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:50.329+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:50.513 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:50.363+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:50.364890+0000) 2024-02-19T20:38:50.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:50.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:50.513 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:50.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:50.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:50.242+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:50.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:50.242+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:50.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:50.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:50.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:50.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:50.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:50.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:50.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:50.679+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:51.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:50.890+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:50.891842+0000) 2024-02-19T20:38:51.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:50.957+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:50.958342+0000) 2024-02-19T20:38:51.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:51.364+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:51.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:51.366+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.367042+0000) 2024-02-19T20:38:51.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:51.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:51.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:51.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:51.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:51.279+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:51.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:51.280+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:51.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:51.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:51.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:51.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:51.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:51.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:51.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:51.673+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:52.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:51.960+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.961770+0000) 2024-02-19T20:38:52.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:51.892+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:51.893950+0000) 2024-02-19T20:38:52.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:52.368+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:52.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:52.368+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:52.369690+0000) 2024-02-19T20:38:52.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:52.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:52.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:52.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:52.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:52.317+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:52.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:52.318+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:52.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:52.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:52.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:52.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:52.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:52.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:52.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:52.680+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:53.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:52.895+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:52.896071+0000) 2024-02-19T20:38:53.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:52.965+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:52.965172+0000) 2024-02-19T20:38:53.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:53.371+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:53.371747+0000) 2024-02-19T20:38:53.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:53.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:53.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:53.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:53.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:53.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:53.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:53.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:53.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:53.317+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:53.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:53.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:53.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:53.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:53.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:53.680+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:53.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:53.681+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:54.101 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:54.110+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69088 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:54.101 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:54.105+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.105193+0000) 2024-02-19T20:38:54.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:53.898+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:53.898280+0000) 2024-02-19T20:38:54.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:54.105+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.105621+0000) 2024-02-19T20:38:54.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:53.967+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:53.967971+0000) 2024-02-19T20:38:54.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:54.107+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.107266+0000) 2024-02-19T20:38:54.357 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:54.340+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:54.358 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:54.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:54.358 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:54.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:54.358 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:54.345+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:54.358 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:54.346+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:54.631 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:54.371+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.372888+0000) 2024-02-19T20:38:54.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:54.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:54.885 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:54.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:54.885 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:54.693+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:54.885 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:54.694+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:55.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:54.898+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.899601+0000) 2024-02-19T20:38:55.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:54.968+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:54.969530+0000) 2024-02-19T20:38:55.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:55.364+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:55.519 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:55.374+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:55.375246+0000) 2024-02-19T20:38:55.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:55.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:55.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:55.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:55.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:55.394+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:55.519 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:55.395+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:55.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:55.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:55.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:55.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:55.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:55.730+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:55.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:55.730+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:56.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:55.900+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:55.901784+0000) 2024-02-19T20:38:56.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:55.971+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:55.972134+0000) 2024-02-19T20:38:56.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:56.335+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:56.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:56.377+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.378676+0000) 2024-02-19T20:38:56.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:56.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:56.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:56.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:56.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:56.396+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:56.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:56.397+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:56.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:56.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:56.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:56.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:56.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:56.699+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:56.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:56.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:57.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:56.973+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.974341+0000) 2024-02-19T20:38:57.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:56.902+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:56.903972+0000) 2024-02-19T20:38:57.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:57.332+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:57.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:57.381+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:57.382164+0000) 2024-02-19T20:38:57.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:57.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:57.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:57.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:57.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:57.400+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:57.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:57.401+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:57.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:57.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:57.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:57.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:57.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:57.679+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:57.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:57.679+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:58.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:57.975+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:57.976631+0000) 2024-02-19T20:38:58.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:57.905+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:57.906444+0000) 2024-02-19T20:38:58.492 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-19T20:38:58.506 INFO:tasks.cephadm:Teardown begin 2024-02-19T20:38:58.506 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-19T20:38:58.513 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-19T20:38:58.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:58.324+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:58.521 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:58.384+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:58.385703+0000) 2024-02-19T20:38:58.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:58.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:58.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:58.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:58.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:58.389+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:58.522 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:58.389+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:58.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:38:58.546 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:38:58.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:58.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:58.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:58.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:58.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:58.642+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:58.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:58.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:59.100 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:59.113+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.114963+0000) 2024-02-19T20:38:59.100 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:58.979+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:58.980263+0000) 2024-02-19T20:38:59.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:58.908+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:58.909164+0000) 2024-02-19T20:38:59.358 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:38:59.118+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69127 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:38:59.358 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:59.114+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.115737+0000) 2024-02-19T20:38:59.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:59.114+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.115852+0000) 2024-02-19T20:38:59.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:59.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:59.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:59.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:59.359 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:59.372+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:38:59.632 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:59.373+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:38:59.632 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:59.373+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:38:59.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:38:59.385+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.386839+0000) 2024-02-19T20:38:59.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:59.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:59.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:59.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:59.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:59.602+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:38:59.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:59.603+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:00.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:38:59.909+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.910301+0000) 2024-02-19T20:39:00.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:38:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:38:59.981+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:38:59.982132+0000) 2024-02-19T20:39:00.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:00.383+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:00.524 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:00.387+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:00.388847+0000) 2024-02-19T20:39:00.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:00.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:00.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:00.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:00.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:00.342+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:00.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:00.343+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:00.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:00.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:00.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:00.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:00.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:00.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:00.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:00.640+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:01.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:00.983+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:00.984215+0000) 2024-02-19T20:39:01.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:00.911+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:00.912538+0000) 2024-02-19T20:39:01.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:01.370+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:01.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:01.390+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.391347+0000) 2024-02-19T20:39:01.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:01.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:01.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:01.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:01.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:01.362+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:01.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:01.363+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:01.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:39:01.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:01.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:01.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:01.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:01.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:01.672+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:01.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:01.672+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:02.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:01.985+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.986787+0000) 2024-02-19T20:39:02.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:01.913+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:01.914710+0000) 2024-02-19T20:39:02.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:02.328+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:02.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:02.392+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:02.393571+0000) 2024-02-19T20:39:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:02.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:02.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:02.390+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:02.391+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:02.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:02.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:02.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:02.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:02.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:02.689+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:26.294525+0000 (oldest deadline 2024-02-19T20:28:46.294525+0000) 2024-02-19T20:39:02.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:02.689+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:03.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:02.915+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:02.916779+0000) 2024-02-19T20:39:03.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:02.988+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:02.989308+0000) 2024-02-19T20:39:03.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:03.371+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:03.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:03.394+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:03.395610+0000) 2024-02-19T20:39:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:03.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:03.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:03.351+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:03.351+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:03.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:03.651+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:04.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:03.990+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:03.991739+0000) 2024-02-19T20:39:04.106 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:03.918+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:03.919342+0000) 2024-02-19T20:39:04.382 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:04.121+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.122864+0000) 2024-02-19T20:39:04.382 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:04.322+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:04.382 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:04.123+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.124345+0000) 2024-02-19T20:39:04.383 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:04.126+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69167 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:04.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:04.122+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.123178+0000) 2024-02-19T20:39:04.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:04.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:04.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:04.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:04.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:04.358+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:04.383 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:04.358+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:04.632 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:04.395+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.397008+0000) 2024-02-19T20:39:04.633 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:04.638+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:05.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:04.919+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.920515+0000) 2024-02-19T20:39:05.277 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:04.991+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:04.992934+0000) 2024-02-19T20:39:05.632 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:05.369+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:05.633 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:05.398+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:05.399557+0000) 2024-02-19T20:39:05.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:05.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:05.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:05.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:05.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:05.384+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:05.633 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:05.384+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:05.908 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:05.649+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:06.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:05.922+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:05.923971+0000) 2024-02-19T20:39:06.277 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:05.993+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:05.994909+0000) 2024-02-19T20:39:06.584 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:06.341+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:06.585 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:06.400+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.401511+0000) 2024-02-19T20:39:06.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:06.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:06.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:06.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:06.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:06.410+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:06.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:06.411+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:06.850 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:06.643+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:07.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:06.996+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.997478+0000) 2024-02-19T20:39:07.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:06.926+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:06.927451+0000) 2024-02-19T20:39:07.404 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:07.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:07.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:07.402+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:07.403518+0000) 2024-02-19T20:39:07.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:07.639+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:07.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:07.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:07.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:07.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:07.779 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:07.419+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:07.780 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:07.419+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:08.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:07.928+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:07.929676+0000) 2024-02-19T20:39:08.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:07.998+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:07.999522+0000) 2024-02-19T20:39:08.450 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:08.294+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:08.450 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:08.404+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:08.405727+0000) 2024-02-19T20:39:08.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:08.687+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:08.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:08.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:08.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:08.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:08.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:08.464+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:08.777 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:08.465+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:09.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:09.000+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.001589+0000) 2024-02-19T20:39:09.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:08.930+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:08.931907+0000) 2024-02-19T20:39:09.391 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:09.134+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69206 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:09.392 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:09.131+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.132311+0000) 2024-02-19T20:39:09.392 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:09.130+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.131782+0000) 2024-02-19T20:39:09.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:09.130+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.131097+0000) 2024-02-19T20:39:09.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:09.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:09.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:09.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:09.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:09.324+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:09.393 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:09.325+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:09.668 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:09.406+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.407047+0000) 2024-02-19T20:39:09.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:09.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:09.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:09.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:09.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:09.446+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:09.669 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:09.447+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:09.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:09.682+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:09.988 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:09.932+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:09.933062+0000) 2024-02-19T20:39:10.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:10.001+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:10.002743+0000) 2024-02-19T20:39:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:10.301+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:10.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:10.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:10.531 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:10.303+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:10.532 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:10.408+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:10.409702+0000) 2024-02-19T20:39:10.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:10.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:10.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:10.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:10.532 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:10.412+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:10.533 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:10.413+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:10.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:39:10.630 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:10.851 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:10.700+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:11.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:10.934+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:10.935068+0000) 2024-02-19T20:39:11.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:11.004+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.005185+0000) 2024-02-19T20:39:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:11.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:11.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:11.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:11.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:11.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:11.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:11.306+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:11.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:11.410+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.411878+0000) 2024-02-19T20:39:11.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:11.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:11.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:11.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:11.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:11.398+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:28:35.069749+0000 (oldest deadline 2024-02-19T20:28:55.069749+0000) 2024-02-19T20:39:11.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:11.399+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:11.851 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:11.691+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:12.149 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:12.006+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:12.007237+0000) 2024-02-19T20:39:12.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:11.936+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:11.937126+0000) 2024-02-19T20:39:12.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:12.411+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:12.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:12.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:12.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:12.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:12.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:12.264+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:12.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:12.264+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:12.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:12.413+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:12.414058+0000) 2024-02-19T20:39:12.851 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:12.681+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:13.149 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:12.938+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:12.939569+0000) 2024-02-19T20:39:13.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:13.008+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:13.009684+0000) 2024-02-19T20:39:13.401 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:13.372+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:13.401 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:13.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:13.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:13.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:13.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:13.255+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:13.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:13.255+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:13.696 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:39:13.716 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:13.415+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:13.416620+0000) 2024-02-19T20:39:13.996 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:13.730+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:13.997 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:13.941+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:13.942176+0000) 2024-02-19T20:39:14.277 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:14.142+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69246 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:14.313 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:14.138+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.139939+0000) 2024-02-19T20:39:14.313 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:14.139+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.140302+0000) 2024-02-19T20:39:14.313 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:14.010+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.011751+0000) 2024-02-19T20:39:14.314 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:14.139+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.140148+0000) 2024-02-19T20:39:14.585 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:14.341+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:14.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:14.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:14.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:14.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:14.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:14.305+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:14.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:14.306+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:14.586 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:14.416+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.417964+0000) 2024-02-19T20:39:14.851 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:14.706+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:15.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:14.943+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:14.944126+0000) 2024-02-19T20:39:15.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:15.012+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:15.013833+0000) 2024-02-19T20:39:15.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:15.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:15.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:15.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:15.405 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:15.265+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:15.406 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:15.265+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:15.406 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:15.318+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:15.719 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:15.419+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:15.420523+0000) 2024-02-19T20:39:16.000 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:15.733+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:16.001 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:15.945+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:15.946134+0000) 2024-02-19T20:39:16.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:16.014+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.015934+0000) 2024-02-19T20:39:16.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:16.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:16.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:16.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:16.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:16.261+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:16.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:16.262+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:16.546 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:16.422+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.423282+0000) 2024-02-19T20:39:16.546 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:16.306+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:16.851 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:16.775+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:17.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:16.947+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:16.948218+0000) 2024-02-19T20:39:17.150 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:17.017+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:17.018688+0000) 2024-02-19T20:39:17.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:17.267+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:17.412 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:17.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:17.412 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:17.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:17.412 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:17.249+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:17.412 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:17.250+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:17.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:17.424+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:17.425998+0000) 2024-02-19T20:39:18.046 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:17.950+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:17.951959+0000) 2024-02-19T20:39:18.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:17.803+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:18.046 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:18.019+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:18.020757+0000) 2024-02-19T20:39:18.412 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:18.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:18.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:18.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:18.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:18.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:18.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:18.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:18.413 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:18.202+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:18.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:18.427+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:18.428084+0000) 2024-02-19T20:39:19.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:18.953+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:18.954251+0000) 2024-02-19T20:39:19.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:18.815+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:19.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:19.021+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.022838+0000) 2024-02-19T20:39:19.414 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:19.152+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69285 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:19.415 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:19.148+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.149232+0000) 2024-02-19T20:39:19.415 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:19.147+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.148851+0000) 2024-02-19T20:39:19.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:19.216+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:19.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:19.146+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.147583+0000) 2024-02-19T20:39:19.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:19.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:19.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:19.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:19.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:19.243+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:19.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:19.245+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:19.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:19.428+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.429473+0000) 2024-02-19T20:39:20.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:19.954+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:19.955995+0000) 2024-02-19T20:39:20.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:19.811+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:20.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:20.023+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:20.024060+0000) 2024-02-19T20:39:20.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:20.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:20.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:20.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:20.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:20.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:20.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:20.218+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:20.417 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:20.219+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:20.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:20.430+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:20.431776+0000) 2024-02-19T20:39:21.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:20.958+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:20.959411+0000) 2024-02-19T20:39:21.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:20.824+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:21.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:21.025+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.026090+0000) 2024-02-19T20:39:21.418 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:21.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:21.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:21.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:21.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:21.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:21.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:21.215+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:21.419 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:21.216+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:21.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:21.432+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.433793+0000) 2024-02-19T20:39:22.150 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:21.961+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:21.962761+0000) 2024-02-19T20:39:22.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:21.829+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:22.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:22.027+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:22.028224+0000) 2024-02-19T20:39:22.422 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:22.253+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:22.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:22.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:22.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:22.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:22.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:22.226+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:22.423 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:22.227+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:22.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:22.436+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:22.437433+0000) 2024-02-19T20:39:23.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:22.965+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:22.966064+0000) 2024-02-19T20:39:23.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:22.848+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:23.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:23.030+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:23.031505+0000) 2024-02-19T20:39:23.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:23.248+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:23.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:23.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:23.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:23.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:23.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:23.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:23.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:23.180+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:23.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:23.439+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:23.440978+0000) 2024-02-19T20:39:24.139 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:23.967+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:23.968064+0000) 2024-02-19T20:39:24.139 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:23.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:24.139 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:23.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:24.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:23.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:24.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:23.897+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:24.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:24.032+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.033672+0000) 2024-02-19T20:39:24.428 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:24.160+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69325 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:24.428 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:24.156+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.157709+0000) 2024-02-19T20:39:24.429 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:24.156+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.157320+0000) 2024-02-19T20:39:24.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:24.231+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:24.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:24.156+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.157721+0000) 2024-02-19T20:39:24.430 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:24.222+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:24.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:24.222+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:24.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:24.222+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:24.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:24.223+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:24.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:24.441+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.442957+0000) 2024-02-19T20:39:25.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:24.968+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:24.969213+0000) 2024-02-19T20:39:25.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:24.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:25.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:24.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:25.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:24.857+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:25.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:24.858+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:25.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:25.034+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:25.035639+0000) 2024-02-19T20:39:25.430 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:25.205+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:25.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:25.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:25.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:25.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:25.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:25.242+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:25.431 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:25.243+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:25.697 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:39:25.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:25.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:25.444+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:25.445509+0000) 2024-02-19T20:39:26.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:25.970+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:25.971541+0000) 2024-02-19T20:39:26.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:25.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:26.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:25.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:26.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:25.906+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:26.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:25.906+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:26.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:26.038+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.039214+0000) 2024-02-19T20:39:26.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:26.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:26.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:26.212+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:26.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:26.212+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:26.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:26.212+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:26.434 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:26.212+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:26.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:26.448+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.449063+0000) 2024-02-19T20:39:27.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:26.974+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:26.975155+0000) 2024-02-19T20:39:27.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:26.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:27.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:26.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:27.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:26.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:27.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:26.890+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:27.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:27.040+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:27.041688+0000) 2024-02-19T20:39:27.435 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:27.177+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:27.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:27.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:27.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:27.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:27.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:27.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:27.436 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:27.180+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:27.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:27.450+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:27.451160+0000) 2024-02-19T20:39:28.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:27.976+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:27.977618+0000) 2024-02-19T20:39:28.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:27.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:28.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:27.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:28.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:27.890+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:28.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:27.890+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:28.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:28.043+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:28.044369+0000) 2024-02-19T20:39:28.437 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:28.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:28.438 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:28.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:28.438 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:28.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:28.438 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:28.202+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:28.438 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:28.203+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:28.768 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:39:28.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:28.452+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:28.453212+0000) 2024-02-19T20:39:29.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:28.979+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:28.980091+0000) 2024-02-19T20:39:29.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:28.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:29.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:28.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:29.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:28.864+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:29.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:28.864+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:29.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:29.046+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.047060+0000) 2024-02-19T20:39:29.439 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:29.169+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69364 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:29.440 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:29.164+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.165666+0000) 2024-02-19T20:39:29.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:29.164+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.165705+0000) 2024-02-19T20:39:29.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:29.221+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:29.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:29.165+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.166576+0000) 2024-02-19T20:39:29.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:29.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:29.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:29.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:29.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:29.223+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:29.441 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:29.224+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:29.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:29.454+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.455231+0000) 2024-02-19T20:39:30.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:29.980+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:29.981712+0000) 2024-02-19T20:39:30.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:29.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:30.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:29.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:30.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:29.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:30.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:29.906+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:30.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:30.047+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:30.048297+0000) 2024-02-19T20:39:30.442 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:30.228+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:30.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:30.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:30.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:30.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:30.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:30.187+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:30.443 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:30.188+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:30.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:30.457+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:30.458235+0000) 2024-02-19T20:39:31.151 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:30.982+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:30.983858+0000) 2024-02-19T20:39:31.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:30.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:31.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:30.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:31.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:30.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:31.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:30.923+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:31.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:31.049+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.050354+0000) 2024-02-19T20:39:31.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:31.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:31.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:31.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:31.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:31.177+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:31.445 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:31.178+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:31.446 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:31.238+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:31.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:39:31.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:31.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:31.459+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.460404+0000) 2024-02-19T20:39:31.840 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:39:31.840 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:39:31.841 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-19T20:39:31.841 DEBUG:teuthology.orchestra.run.smithi077:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-19T20:39:31.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:39:31.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:32.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:31.985+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:31.986358+0000) 2024-02-19T20:39:32.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:31.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:32.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:31.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:32.130 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:31.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:32.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:31.953+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:32.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:32.052+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:32.053037+0000) 2024-02-19T20:39:32.447 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:32.202+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:32.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:32.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:32.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:32.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:32.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:32.144+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:32.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:32.145+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:32.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:32.461+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:32.462998+0000) 2024-02-19T20:39:33.141 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:32.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:33.141 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:32.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:33.141 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:32.953+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:33.141 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:32.954+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:33.141 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:33.055+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:33.056729+0000) 2024-02-19T20:39:33.142 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:32.988+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:32.989994+0000) 2024-02-19T20:39:33.451 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:33.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:33.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:33.155+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:33.451 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:33.155+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:33.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:33.155+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:33.452 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:33.156+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:33.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:33.465+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:33.466499+0000) 2024-02-19T20:39:34.137 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:33.991+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:33.992838+0000) 2024-02-19T20:39:34.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:33.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:34.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:33.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:34.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:33.931+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:34.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:33.932+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:34.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:34.058+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.059273+0000) 2024-02-19T20:39:34.453 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:34.177+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69404 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:34.453 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:34.173+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.174934+0000) 2024-02-19T20:39:34.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:34.173+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.174172+0000) 2024-02-19T20:39:34.454 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:34.196+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:34.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:34.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:34.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:34.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:34.454 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:34.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:34.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:34.153+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:34.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:34.174+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.175139+0000) 2024-02-19T20:39:34.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:34.467+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.468414+0000) 2024-02-19T20:39:34.940 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:39:35.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:34.993+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:34.994067+0000) 2024-02-19T20:39:35.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:34.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:35.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:34.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:35.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:34.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:35.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:34.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:35.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:35.059+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:35.060749+0000) 2024-02-19T20:39:35.455 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:35.164+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:35.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:35.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:35.455 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:35.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:35.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:35.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:35.456 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:35.136+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:35.777 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:35.469+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:35.470699+0000) 2024-02-19T20:39:36.105 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:35.996+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:35.997660+0000) 2024-02-19T20:39:36.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:35.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:36.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:35.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:36.105 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:35.952+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:36.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:35.953+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:36.106 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:36.061+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.062744+0000) 2024-02-19T20:39:36.457 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:36.183+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:36.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:36.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:36.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:36.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:36.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:36.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:36.458 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:36.120+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:36.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:36.471+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:36.472831+0000) 2024-02-19T20:39:37.138 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:37.000+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:37.001268+0000) 2024-02-19T20:39:37.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:36.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:37.138 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:36.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:37.139 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:36.960+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:37.139 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:36.961+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:37.139 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:37.063+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:37.064841+0000) 2024-02-19T20:39:37.459 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:37.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:37.460 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:37.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:37.460 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:37.152+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:37.460 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:37.153+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:37.460 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:37.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:37.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:37.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:37.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:37.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:37.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:37.204+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:37.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:37.474+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:37.475045+0000) 2024-02-19T20:39:38.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:38.003+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:38.004773+0000) 2024-02-19T20:39:38.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:38.106+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:38.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:38.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:38.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:38.107+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:38.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:38.107+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:38.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:37.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:38.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:37.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:38.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:37.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:38.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:37.916+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:38.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:38.066+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:38.067381+0000) 2024-02-19T20:39:38.461 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:38.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:38.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:38.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:38.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:38.211+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:38.462 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:38.212+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:38.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:38.476+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:38.477240+0000) 2024-02-19T20:39:39.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:39.007+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.008288+0000) 2024-02-19T20:39:39.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:38.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:39.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:38.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:39.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:38.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:39.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:38.905+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:39.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:39.069+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.070012+0000) 2024-02-19T20:39:39.463 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:39.181+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.182109+0000) 2024-02-19T20:39:39.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:39.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:39.463 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:39.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:39.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:39.136+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:39.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:39.137+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:39.464 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:39.181+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.182448+0000) 2024-02-19T20:39:39.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:39.181+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.182208+0000) 2024-02-19T20:39:39.464 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:39.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:39.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:39.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:39.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:39.222+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:39.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:39.222+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:39.465 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:39.185+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69443 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:39.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:39.477+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:39.478595+0000) 2024-02-19T20:39:40.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:40.010+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:40.010319+0000) 2024-02-19T20:39:40.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:39.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:40.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:39.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:40.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:39.919+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:40.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:39.920+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:40.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:40.071+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:40.071328+0000) 2024-02-19T20:39:40.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:40.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:40.466 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:40.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:40.467 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:40.183+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:40.467 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:40.183+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:40.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:40.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:40.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:40.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:40.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:40.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:40.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:40.186+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:40.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:40.481+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:40.481953+0000) 2024-02-19T20:39:41.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:41.011+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.012821+0000) 2024-02-19T20:39:41.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:40.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:41.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:40.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:41.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:40.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:41.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:40.889+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:41.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:41.072+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.073474+0000) 2024-02-19T20:39:41.469 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:41.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:41.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:41.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:41.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:41.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:41.470 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:41.180+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:41.470 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:41.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:41.470 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:41.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:41.470 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:41.178+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:41.471 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:41.179+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:41.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:41.483+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:41.484929+0000) 2024-02-19T20:39:42.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:42.015+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:42.016499+0000) 2024-02-19T20:39:42.124 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:41.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:42.124 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:41.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:42.124 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:41.905+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:42.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:41.906+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:42.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:42.076+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:42.077100+0000) 2024-02-19T20:39:42.472 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:42.209+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:42.472 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:42.209+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:42.472 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:42.209+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:42.473 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:42.210+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:42.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:42.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:42.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:42.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:42.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:42.137+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:42.473 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:42.138+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:42.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:42.486+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:42.487666+0000) 2024-02-19T20:39:43.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:43.018+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:43.019087+0000) 2024-02-19T20:39:43.121 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:43.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:43.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:42.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:43.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:42.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:43.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:42.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:43.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:42.898+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:43.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:43.078+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:43.079633+0000) 2024-02-19T20:39:43.474 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:43.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:43.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:43.135+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:43.475 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:43.136+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:43.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:43.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:43.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:43.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:43.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:43.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:43.475 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:43.179+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:43.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:43.488+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:43.489973+0000) 2024-02-19T20:39:43.941 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:39:43.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:44.152 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:44.020+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.021318+0000) 2024-02-19T20:39:44.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:43.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:44.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:43.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:44.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:43.902+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:44.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:43.902+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:44.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:44.081+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.082293+0000) 2024-02-19T20:39:44.475 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:44.193+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69483 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:44.476 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:44.188+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.189953+0000) 2024-02-19T20:39:44.476 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:44.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:44.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:44.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:44.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:44.174+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:44.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:44.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:44.477 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:44.189+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.190548+0000) 2024-02-19T20:39:44.477 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:44.189+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.190423+0000) 2024-02-19T20:39:44.477 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:44.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:44.477 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:44.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:44.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:44.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:44.478 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:44.217+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:44.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:44.490+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:44.491292+0000) 2024-02-19T20:39:45.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:45.022+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:45.023375+0000) 2024-02-19T20:39:45.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:44.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:45.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:44.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:45.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:44.913+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:45.153 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:44.913+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:45.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:45.083+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:45.084102+0000) 2024-02-19T20:39:45.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:45.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:45.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:45.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:45.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:45.190+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:45.478 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:45.190+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:45.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:45.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:45.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:45.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:45.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:45.200+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:45.479 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:45.201+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:45.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:45.492+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:45.493486+0000) 2024-02-19T20:39:46.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:46.024+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.025441+0000) 2024-02-19T20:39:46.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:46.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:46.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:46.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:46.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:46.161+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:46.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:45.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:46.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:45.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:46.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:45.932+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:46.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:45.932+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:46.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:46.085+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.086635+0000) 2024-02-19T20:39:46.480 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:46.162+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:46.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:46.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:46.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:46.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:46.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:46.194+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:46.481 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:46.194+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:46.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:46.495+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:46.496147+0000) 2024-02-19T20:39:47.008 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:39:47.117 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:47.026+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:47.027547+0000) 2024-02-19T20:39:47.117 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:46.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:47.117 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:46.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:47.118 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:46.966+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:47.118 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:46.966+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:47.118 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:47.087+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:47.088787+0000) 2024-02-19T20:39:47.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:47.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:47.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:47.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:47.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:47.148+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:47.485 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:47.149+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:47.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:47.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:47.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:47.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:47.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:47.131+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:47.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:47.132+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:47.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:47.498+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:47.499747+0000) 2024-02-19T20:39:48.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:48.030+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:48.031204+0000) 2024-02-19T20:39:48.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:48.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:48.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:48.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:48.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:48.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:48.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:48.110+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:48.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:47.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:48.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:47.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:48.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:47.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:48.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:47.928+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:48.120 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:48.089+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:48.090969+0000) 2024-02-19T20:39:48.487 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:48.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:48.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:48.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:48.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:48.132+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:48.488 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:48.133+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:48.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:48.502+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:48.503257+0000) 2024-02-19T20:39:49.140 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:49.114+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:49.140 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:49.114+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:49.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:49.114+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:49.141 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:49.114+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:49.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:49.032+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.033258+0000) 2024-02-19T20:39:49.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:49.154+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:49.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:49.154+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:49.142 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:49.154+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:49.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:48.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:49.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:48.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:49.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:48.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:49.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:48.933+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:49.143 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:49.092+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.093457+0000) 2024-02-19T20:39:49.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:49.201+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69522 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:49.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:49.198+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.199653+0000) 2024-02-19T20:39:49.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:49.198+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.199204+0000) 2024-02-19T20:39:49.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:49.155+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:49.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:49.198+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.199125+0000) 2024-02-19T20:39:49.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:49.503+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:49.504860+0000) 2024-02-19T20:39:50.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:50.033+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:50.034797+0000) 2024-02-19T20:39:50.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:49.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:50.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:49.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:50.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:49.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:50.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:49.959+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:50.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:50.093+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:50.094730+0000) 2024-02-19T20:39:50.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:50.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:50.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:50.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:50.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:50.090+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:50.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:50.091+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:50.465 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:50.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:50.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:50.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:50.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:50.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:50.466 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:50.203+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:50.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:50.506+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:50.507077+0000) 2024-02-19T20:39:51.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:51.035+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.036876+0000) 2024-02-19T20:39:51.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:51.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:51.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:51.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:51.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:51.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:51.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:51.111+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:51.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:50.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:51.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:50.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:51.156 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:50.933+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:51.156 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:50.934+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:51.156 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:51.095+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.096853+0000) 2024-02-19T20:39:51.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:51.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:51.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:51.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:51.494 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:51.240+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:51.495 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:51.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:51.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:51.508+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:51.509713+0000) 2024-02-19T20:39:52.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:52.039+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:52.040520+0000) 2024-02-19T20:39:52.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:52.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:52.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:52.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:52.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:52.072+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:52.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:52.073+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:52.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:51.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:52.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:51.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:52.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:51.959+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:52.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:51.960+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:52.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:52.098+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:52.099085+0000) 2024-02-19T20:39:52.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:52.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:52.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:52.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:52.496 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:52.208+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:52.497 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:52.209+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:52.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:52.510+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:52.511850+0000) 2024-02-19T20:39:53.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:53.041+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:53.042541+0000) 2024-02-19T20:39:53.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:52.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:53.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:52.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:53.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:52.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:53.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:52.915+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:53.154 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:53.100+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:53.101123+0000) 2024-02-19T20:39:53.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:53.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:53.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:53.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:53.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:53.064+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:53.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:53.066+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:53.498 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:53.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:53.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:53.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:53.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:53.243+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:53.499 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:53.243+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:53.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:53.512+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:53.513991+0000) 2024-02-19T20:39:54.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:54.043+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.044619+0000) 2024-02-19T20:39:54.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:54.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:54.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:54.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:54.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:54.030+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:54.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:54.031+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:54.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:53.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:54.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:53.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:54.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:53.897+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:54.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:53.898+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:54.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:54.102+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.103806+0000) 2024-02-19T20:39:54.499 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:54.210+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69562 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:54.500 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:54.206+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.207826+0000) 2024-02-19T20:39:54.500 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:54.206+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.207592+0000) 2024-02-19T20:39:54.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:54.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:54.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:54.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:54.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:54.202+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:54.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:54.203+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:54.501 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:54.206+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.207233+0000) 2024-02-19T20:39:54.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:54.514+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:54.515264+0000) 2024-02-19T20:39:55.154 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:55.044+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:55.045857+0000) 2024-02-19T20:39:55.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:55.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:55.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:55.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:55.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:55.077+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:55.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:55.078+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:55.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:54.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:55.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:54.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:55.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:54.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:55.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:54.876+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:55.156 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:55.104+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:55.105033+0000) 2024-02-19T20:39:55.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:55.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:55.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:55.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:55.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:55.170+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:55.502 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:55.171+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:55.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:55.516+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:55.517454+0000) 2024-02-19T20:39:56.091 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:56.047+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.048324+0000) 2024-02-19T20:39:56.092 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:55.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:56.092 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:55.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:56.092 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:55.845+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:56.092 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:55.846+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:56.093 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:56.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:56.093 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:56.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:56.093 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:56.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:56.093 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:56.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:56.504 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:56.105+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.107006+0000) 2024-02-19T20:39:56.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:56.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:56.504 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:56.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:56.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:56.175+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:56.505 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:56.175+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:56.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:56.518+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:56.519677+0000) 2024-02-19T20:39:57.154 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:57.049+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:57.050371+0000) 2024-02-19T20:39:57.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:57.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:57.154 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:57.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:57.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:57.094+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:57.155 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:57.094+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:57.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:56.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:57.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:56.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:57.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:56.879+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:57.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:56.880+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:57.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:57.108+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:57.109618+0000) 2024-02-19T20:39:57.507 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:57.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:57.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:57.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:57.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:57.193+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:57.508 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:57.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:57.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:57.522+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:57.523241+0000) 2024-02-19T20:39:58.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:58.051+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:58.052466+0000) 2024-02-19T20:39:58.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:57.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:58.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:57.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:58.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:57.878+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:58.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:57.878+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:58.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:58.111+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:58.112065+0000) 2024-02-19T20:39:58.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:58.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:58.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:58.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:58.511 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:58.141+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:58.512 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:58.142+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:58.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:58.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:58.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:58.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:58.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:58.212+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:58.512 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:58.213+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:58.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:58.525+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:58.526833+0000) 2024-02-19T20:39:59.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:39:59.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:39:59.154 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:59.053+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.054530+0000) 2024-02-19T20:39:59.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:58.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:59.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:58.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:59.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:58.898+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:39:59.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:58.899+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:59.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:59.114+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.115369+0000) 2024-02-19T20:39:59.483 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:59.214+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.215908+0000) 2024-02-19T20:39:59.484 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:39:59.219+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69602 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:39:59.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:59.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:59.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:59.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:59.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:59.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:39:59.485 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:59.180+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:39:59.485 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:59.214+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.215523+0000) 2024-02-19T20:39:59.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:59.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:59.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:59.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:59.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:59.188+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:39:59.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:59.189+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:39:59.485 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:39:59.214+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.215271+0000) 2024-02-19T20:39:59.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:39:59.527+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:39:59.528489+0000) 2024-02-19T20:40:00.101 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:00.055+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:00.056565+0000) 2024-02-19T20:40:00.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:59.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:00.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:59.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:00.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:59.855+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:00.102 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:39:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:39:59.856+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:00.515 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:00.115+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:00.116891+0000) 2024-02-19T20:40:00.515 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:00.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:00.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:00.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:00.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:00.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:00.516 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:00.134+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:00.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:00.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:00.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:00.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:00.516 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:00.186+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:00.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:00.186+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:00.778 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:00.529+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:00.530680+0000) 2024-02-19T20:40:01.154 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:01.058+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.059878+0000) 2024-02-19T20:40:01.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:00.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:01.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:00.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:01.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:00.888+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:01.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:00.889+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:01.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:01.118+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.119107+0000) 2024-02-19T20:40:01.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:01.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:01.517 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:01.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:01.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:01.183+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:01.518 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:01.183+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:01.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:01.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:01.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:01.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:01.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:01.181+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:01.518 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:01.182+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:01.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:01.531+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:01.532991+0000) 2024-02-19T20:40:02.076 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:40:02.139 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:02.060+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:02.061975+0000) 2024-02-19T20:40:02.139 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:01.926+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:02.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:01.926+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:02.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:01.926+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:02.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:01.927+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:02.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:02.120+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:02.121479+0000) 2024-02-19T20:40:02.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:02.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:02.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:02.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:02.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:02.168+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:02.520 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:02.169+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:02.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:02.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:02.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:02.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:02.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:02.153+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:02.521 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:02.154+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:02.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:02.534+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:02.535534+0000) 2024-02-19T20:40:03.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:02.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:03.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:02.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:03.109 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:02.930+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:03.109 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:02.931+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:03.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:03.064+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:03.065636+0000) 2024-02-19T20:40:03.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:03.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:03.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:03.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:03.109 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:03.116+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:03.110 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:03.117+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:03.522 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:03.122+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:03.123895+0000) 2024-02-19T20:40:03.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:03.205+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:03.522 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:03.205+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:03.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:03.205+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:03.523 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:03.206+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:03.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:03.536+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:03.537729+0000) 2024-02-19T20:40:04.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:03.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:04.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:03.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:04.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:03.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:04.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:03.935+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:04.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:04.068+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.069243+0000) 2024-02-19T20:40:04.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:04.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:04.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:04.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:04.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:04.077+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:04.113 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:04.077+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:04.524 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:04.227+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69641 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:04.524 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:04.223+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.224015+0000) 2024-02-19T20:40:04.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:04.125+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.126585+0000) 2024-02-19T20:40:04.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:04.223+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.224682+0000) 2024-02-19T20:40:04.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:04.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:04.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:04.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:04.525 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:04.217+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:04.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:04.218+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:04.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:04.223+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.224384+0000) 2024-02-19T20:40:04.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:04.538+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:04.539788+0000) 2024-02-19T20:40:05.077 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:40:05.078 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:05.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:04.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:05.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:04.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:05.113 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:04.956+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:05.114 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:04.956+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:05.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:05.070+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:05.071378+0000) 2024-02-19T20:40:05.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:05.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:05.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:05.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:05.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:05.102+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:05.114 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:05.103+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:05.152 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:40:05.152 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:40:05.153 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-19T20:40:05.155 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617/remote/smithi077/crash 2024-02-19T20:40:05.156 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/crash -- . 2024-02-19T20:40:05.184 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617/remote/smithi104/crash 2024-02-19T20:40:05.185 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:40:05.185 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:05.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:05.127+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:05.128178+0000) 2024-02-19T20:40:05.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:05.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:05.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:05.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:05.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:05.179+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:05.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:05.179+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:05.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:05.540+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:05.541937+0000) 2024-02-19T20:40:06.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:05.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:06.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:05.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:06.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:05.936+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:06.115 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:05.937+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:06.116 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:06.072+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.073433+0000) 2024-02-19T20:40:06.116 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:06.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:06.116 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:06.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:06.116 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:06.079+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:06.116 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:06.079+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:06.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:06.129+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.130739+0000) 2024-02-19T20:40:06.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:06.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:06.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:06.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:06.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:06.156+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:06.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:06.156+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:06.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:06.543+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:06.544159+0000) 2024-02-19T20:40:07.117 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:06.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:07.117 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:06.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:07.117 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:06.922+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:07.117 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:06.922+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:07.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:07.074+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:07.075475+0000) 2024-02-19T20:40:07.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:07.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:07.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:07.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:07.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:07.109+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:07.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:07.109+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:07.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:07.131+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:07.132843+0000) 2024-02-19T20:40:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:07.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:07.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:07.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:07.159+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:07.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:07.160+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:07.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:07.545+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:07.546172+0000) 2024-02-19T20:40:08.110 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:07.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:08.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:07.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:08.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:07.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:08.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:07.874+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:08.111 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:08.076+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:08.077591+0000) 2024-02-19T20:40:08.111 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:08.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:08.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:08.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:08.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:08.125+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:08.252 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:40:08.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:08.134+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:08.135075+0000) 2024-02-19T20:40:08.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:08.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:08.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:08.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:08.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:08.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:08.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:08.158+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:08.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:08.159+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:08.856 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:08.547+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:08.548185+0000) 2024-02-19T20:40:09.121 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:08.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:09.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:08.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:09.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:08.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:09.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:08.899+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:09.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:09.078+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.079682+0000) 2024-02-19T20:40:09.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:09.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:09.122 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:09.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:09.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:09.089+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:09.123 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:09.089+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:09.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:09.235+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69681 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:09.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:09.230+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.231909+0000) 2024-02-19T20:40:09.483 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:09.136+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.137204+0000) 2024-02-19T20:40:09.483 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:09.231+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.232381+0000) 2024-02-19T20:40:09.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:09.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:09.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:09.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:09.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:09.157+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:09.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:09.158+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:09.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:09.231+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.232211+0000) 2024-02-19T20:40:09.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:09.548+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:09.549460+0000) 2024-02-19T20:40:10.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:09.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:10.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:09.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:10.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:09.889+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:10.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:09.889+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:10.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:10.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:10.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:10.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:10.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:10.043+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:10.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:10.044+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:10.124 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:10.079+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:10.080970+0000) 2024-02-19T20:40:10.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:10.137+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:10.138550+0000) 2024-02-19T20:40:10.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:10.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:10.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:10.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:10.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:10.170+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:10.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:10.170+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:10.836 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:10.550+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:10.551536+0000) 2024-02-19T20:40:11.107 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:10.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:11.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:10.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:11.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:10.851+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:11.108 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:10.851+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:11.111 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:11.082+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.083417+0000) 2024-02-19T20:40:11.111 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:11.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:11.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:11.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:11.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:11.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:11.112 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:11.084+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:11.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:11.140+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.141062+0000) 2024-02-19T20:40:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:11.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:11.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:11.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:11.122+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:11.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:11.123+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:11.847 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:11.554+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:11.555062+0000) 2024-02-19T20:40:12.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:11.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:12.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:11.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:12.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:11.861+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:12.101 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:11.862+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:12.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:12.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:12.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:12.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:12.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:12.047+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:12.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:12.048+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:12.102 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:12.084+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:12.085826+0000) 2024-02-19T20:40:12.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:12.142+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:12.143068+0000) 2024-02-19T20:40:12.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:12.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:12.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:12.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:12.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:12.115+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:12.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:12.116+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:12.840 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:12.556+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:12.557061+0000) 2024-02-19T20:40:13.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:13.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:13.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:13.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:13.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:13.069+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:13.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:13.070+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:13.130 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:13.087+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:13.088527+0000) 2024-02-19T20:40:13.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:12.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:13.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:12.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:13.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:12.854+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:13.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:12.855+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:13.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:13.144+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:13.145487+0000) 2024-02-19T20:40:13.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:13.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:13.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:13.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:13.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:13.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:13.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:13.161+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:13.800 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:13.558+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:13.559150+0000) 2024-02-19T20:40:14.051 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:13.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:14.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:13.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:14.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:13.814+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:14.052 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:13.815+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:14.346 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:14.243+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69720 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:14.346 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:14.148+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.149177+0000) 2024-02-19T20:40:14.346 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:14.239+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.240353+0000) 2024-02-19T20:40:14.347 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:14.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:14.347 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:14.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:14.347 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:14.180+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:14.347 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:14.180+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:14.347 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:14.239+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.240871+0000) 2024-02-19T20:40:14.347 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:14.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:14.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:14.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:14.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:14.081+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:14.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:14.082+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:14.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:14.089+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.090662+0000) 2024-02-19T20:40:14.348 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:14.238+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.239976+0000) 2024-02-19T20:40:14.610 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:14.559+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:14.560410+0000) 2024-02-19T20:40:15.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:14.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:15.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:14.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:15.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:14.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:15.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:14.802+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:15.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:15.149+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:15.150491+0000) 2024-02-19T20:40:15.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:15.091+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:15.092048+0000) 2024-02-19T20:40:15.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:15.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:15.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:15.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:15.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:15.130+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:15.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:15.130+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:15.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:15.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:15.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:15.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:15.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:15.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:15.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:15.177+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:15.857 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:15.561+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:15.562365+0000) 2024-02-19T20:40:15.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:15.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:15.884 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:15.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:15.885 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:15.806+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:15.885 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:15.806+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:16.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:16.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:16.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:16.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:16.127 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:16.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:16.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:16.087+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:16.128 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:16.093+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.094242+0000) 2024-02-19T20:40:16.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:16.151+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.152553+0000) 2024-02-19T20:40:16.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:16.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:16.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:16.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:16.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:16.142+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:16.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:16.142+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:16.819 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:16.563+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:16.564572+0000) 2024-02-19T20:40:17.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:16.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:17.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:16.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:17.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:16.834+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:17.073 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:16.834+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:17.255 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:40:17.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:17.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:17.153+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:17.154636+0000) 2024-02-19T20:40:17.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:17.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:17.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:17.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:17.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:17.146+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:17.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:17.147+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:17.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:17.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:17.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:17.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:17.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:17.087+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:17.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:17.088+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:17.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:17.095+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:17.096286+0000) 2024-02-19T20:40:17.845 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:17.565+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:17.566603+0000) 2024-02-19T20:40:18.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:17.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:18.098 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:17.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:18.099 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:17.859+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:18.099 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:17.860+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:18.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:18.097+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:18.098764+0000) 2024-02-19T20:40:18.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:18.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:18.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:18.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:18.099 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:18.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:18.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:18.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:18.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:18.157+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:18.158321+0000) 2024-02-19T20:40:18.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:18.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:18.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:18.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:18.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:18.119+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:18.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:18.119+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:18.857 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:18.567+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:18.568641+0000) 2024-02-19T20:40:19.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:18.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:19.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:18.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:19.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:18.875+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:19.128 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:18.876+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:19.128 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:19.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:19.128 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:19.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:19.128 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:19.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:19.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:19.074+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:19.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:19.099+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.100851+0000) 2024-02-19T20:40:19.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:19.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:19.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:19.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:19.129 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:19.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:19.483 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:19.251+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69760 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:19.483 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:19.248+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.249435+0000) 2024-02-19T20:40:19.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:19.159+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.160352+0000) 2024-02-19T20:40:19.484 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:19.247+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.248259+0000) 2024-02-19T20:40:19.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:19.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:19.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:19.248+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.249078+0000) 2024-02-19T20:40:19.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:19.568+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:19.569931+0000) 2024-02-19T20:40:20.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:20.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:20.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:20.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:20.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:20.076+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:20.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:20.077+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:20.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:19.873+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:20.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:19.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:20.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:19.874+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:20.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:19.874+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:20.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:20.101+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:20.102170+0000) 2024-02-19T20:40:20.147 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:20.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:20.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:20.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:20.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:20.115+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:20.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:20.115+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:20.316 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:40:20.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:20.160+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:20.161692+0000) 2024-02-19T20:40:20.815 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:20.571+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:20.572373+0000) 2024-02-19T20:40:21.088 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:20.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:21.089 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:20.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:21.089 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:20.830+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:21.089 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:20.830+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:21.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:21.162+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.163756+0000) 2024-02-19T20:40:21.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:21.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:21.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:21.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:21.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:21.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:21.104+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:21.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:21.103+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.104212+0000) 2024-02-19T20:40:21.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:21.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:21.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:21.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:21.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:21.124+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:21.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:21.126+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:21.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:21.573+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:21.574778+0000) 2024-02-19T20:40:21.858 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:21.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:21.858 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:21.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:21.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:21.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:21.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:21.818+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:22.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:22.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:22.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:22.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:22.118 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:22.103+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:22.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:22.104+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:22.119 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:22.106+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:22.107798+0000) 2024-02-19T20:40:22.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:22.164+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:22.165811+0000) 2024-02-19T20:40:22.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:22.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:22.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:22.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:22.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:22.133+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:22.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:22.133+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:22.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:22.576+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:22.577499+0000) 2024-02-19T20:40:22.858 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:22.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:22.858 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:22.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:22.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:22.782+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:22.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:22.783+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:23.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:23.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:23.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:23.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:23.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:23.113+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:23.153 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:23.114+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:23.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:23.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:23.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:23.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:23.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:23.088+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:23.153 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:23.089+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:23.154 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:23.110+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:23.111434+0000) 2024-02-19T20:40:23.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:23.167+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:23.168045+0000) 2024-02-19T20:40:23.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:23.579+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:23.580128+0000) 2024-02-19T20:40:23.858 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:23.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:23.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:23.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:23.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:23.790+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:23.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:23.791+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:24.145 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:24.112+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.113506+0000) 2024-02-19T20:40:24.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:24.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:24.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:24.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:24.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:24.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:24.146 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:24.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:24.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:24.256+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.257139+0000) 2024-02-19T20:40:24.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:24.260+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69799 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:24.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:24.170+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.171623+0000) 2024-02-19T20:40:24.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:24.255+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.256179+0000) 2024-02-19T20:40:24.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:24.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:24.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:24.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:24.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:24.160+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:24.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:24.161+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:24.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:24.255+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.256752+0000) 2024-02-19T20:40:24.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:24.580+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:24.581556+0000) 2024-02-19T20:40:24.858 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:24.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:24.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:24.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:24.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:24.802+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:24.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:24.802+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:25.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:25.113+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:25.114840+0000) 2024-02-19T20:40:25.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:25.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:25.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:25.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:25.131 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:25.146+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:25.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:25.146+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:25.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:25.172+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:25.173884+0000) 2024-02-19T20:40:25.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:25.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:25.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:25.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:25.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:25.164+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:25.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:25.165+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:25.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:25.584+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:25.585042+0000) 2024-02-19T20:40:25.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:25.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:25.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:25.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:25.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:25.760+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:25.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:25.761+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:26.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:26.115+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.116915+0000) 2024-02-19T20:40:26.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:26.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:26.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:26.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:26.158 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:26.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:26.158 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:26.119+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:26.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:26.175+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.176408+0000) 2024-02-19T20:40:26.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:26.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:26.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:26.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:26.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:26.173+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:26.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:26.174+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:26.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:26.587+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:26.587432+0000) 2024-02-19T20:40:26.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:26.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:26.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:26.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:26.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:26.786+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:26.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:26.787+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:27.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:27.119+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:27.119134+0000) 2024-02-19T20:40:27.126 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:27.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:27.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:27.178+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:27.178817+0000) 2024-02-19T20:40:27.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:27.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:27.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:27.142+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:27.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:27.142+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:27.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:27.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:27.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:27.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:27.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:27.197+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:27.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:27.197+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:27.858 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:27.588+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:27.589940+0000) 2024-02-19T20:40:27.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:27.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:27.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:27.817+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:27.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:27.818+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:28.157 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:28.120+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:28.121151+0000) 2024-02-19T20:40:28.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:28.180+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:28.181382+0000) 2024-02-19T20:40:28.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:28.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:28.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:28.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:28.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:28.239+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:28.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:28.240+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:28.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:28.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:28.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:28.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:28.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:28.179+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:28.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:28.180+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:28.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:28.591+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:28.592067+0000) 2024-02-19T20:40:28.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:28.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:28.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:28.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:28.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:28.771+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:28.859 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:28.772+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:29.159 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:29.122+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.123694+0000) 2024-02-19T20:40:29.482 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:29.268+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69840 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:29.483 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:29.182+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.183716+0000) 2024-02-19T20:40:29.483 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:29.263+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.264858+0000) 2024-02-19T20:40:29.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:29.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:29.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:29.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:29.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:29.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:29.483 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:29.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:29.484 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:29.264+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.265539+0000) 2024-02-19T20:40:29.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:29.264+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.265793+0000) 2024-02-19T20:40:29.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:29.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:29.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:29.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:29.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:29.279+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:29.484 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:29.280+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:29.756 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:29.592+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:29.593531+0000) 2024-02-19T20:40:30.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:29.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:30.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:29.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:30.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:29.770+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:30.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:29.771+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:30.401 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:30.184+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:30.185316+0000) 2024-02-19T20:40:30.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:30.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:30.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:30.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:30.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:30.247+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:30.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:30.248+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:30.402 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:30.124+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:30.125459+0000) 2024-02-19T20:40:30.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:30.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:30.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:30.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:30.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:30.207+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:30.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:30.208+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:30.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:30.595+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:30.597008+0000) 2024-02-19T20:40:31.111 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:30.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:31.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:30.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:31.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:30.820+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:31.112 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:30.820+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:31.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:31.187+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.188901+0000) 2024-02-19T20:40:31.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:31.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:31.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:31.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:31.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:31.235+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:31.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:31.236+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:31.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:31.126+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.127475+0000) 2024-02-19T20:40:31.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:31.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:31.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:31.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:31.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:31.166+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:31.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:31.167+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:31.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:31.598+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:31.599123+0000) 2024-02-19T20:40:31.860 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:31.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:31.860 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:31.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:31.860 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:31.842+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:31.861 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:31.843+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:32.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:40:32.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:32.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:32.189+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:32.190976+0000) 2024-02-19T20:40:32.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:32.128+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:32.129661+0000) 2024-02-19T20:40:32.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:32.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:32.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:32.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:32.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:32.156+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:32.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:32.157+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:32.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:32.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:32.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:32.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:32.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:32.286+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:32.528 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:32.286+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:32.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:32.601+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:32.602629+0000) 2024-02-19T20:40:33.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:32.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:33.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:32.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:33.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:32.884+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:33.116 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:32.885+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:33.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:33.192+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:33.193054+0000) 2024-02-19T20:40:33.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:33.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:33.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:33.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:33.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:33.304+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:33.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:33.305+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:33.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:33.130+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:33.131855+0000) 2024-02-19T20:40:33.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:33.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:33.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:33.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:33.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:33.177+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:33.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:33.177+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:33.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:33.605+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:33.606122+0000) 2024-02-19T20:40:34.118 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:33.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:34.118 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:33.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:34.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:33.934+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:34.119 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:33.935+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:34.380 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:34.276+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69879 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:34.380 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:34.194+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.195512+0000) 2024-02-19T20:40:34.381 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:34.272+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.273629+0000) 2024-02-19T20:40:34.381 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:34.272+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.273891+0000) 2024-02-19T20:40:34.381 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:34.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:34.381 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:34.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:34.381 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:34.295+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:34.381 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:34.296+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:34.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:34.133+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.134018+0000) 2024-02-19T20:40:34.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:34.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:34.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:34.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:34.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:34.225+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:34.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:34.225+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:34.382 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:34.272+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.273754+0000) 2024-02-19T20:40:34.630 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:34.606+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:34.607454+0000) 2024-02-19T20:40:35.182 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:35.134+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:35.135616+0000) 2024-02-19T20:40:35.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:34.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:35.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:34.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:35.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:34.942+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:35.182 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:34.942+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:35.392 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:40:35.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:35.196+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:35.197715+0000) 2024-02-19T20:40:35.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:35.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:35.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:35.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:35.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:35.234+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:35.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:35.235+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:35.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:35.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:35.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:35.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:35.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:35.337+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:35.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:35.338+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:35.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:35.609+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:35.610896+0000) 2024-02-19T20:40:36.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:35.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:36.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:35.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:36.122 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:35.896+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:36.123 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:35.897+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:36.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:36.200+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.201131+0000) 2024-02-19T20:40:36.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:36.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:36.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:36.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:36.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:36.312+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:36.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:36.313+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:36.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:36.136+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.137654+0000) 2024-02-19T20:40:36.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:36.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:36.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:36.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:36.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:36.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:36.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:36.241+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:36.859 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:36.612+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:36.613055+0000) 2024-02-19T20:40:37.124 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:36.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:37.124 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:36.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:37.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:36.921+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:37.125 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:36.921+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:37.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:37.202+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:37.203311+0000) 2024-02-19T20:40:37.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:37.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:37.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:37.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:37.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:37.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:37.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:37.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:37.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:37.139+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:37.140327+0000) 2024-02-19T20:40:37.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:37.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:37.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:37.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:37.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:37.256+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:37.529 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:37.257+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:37.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:37.614+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:37.615628+0000) 2024-02-19T20:40:38.126 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:37.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:38.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:37.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:38.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:37.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:38.127 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:37.900+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:38.393 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:40:38.394 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:38.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:40:38.460 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:40:38.461 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-19T20:40:38.461 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-02-19T20:40:38.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:38.204+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:38.205997+0000) 2024-02-19T20:40:38.487 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:38.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:38.487 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:38.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:38.487 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:38.285+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:38.487 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:38.285+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:38.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:38.141+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:38.142516+0000) 2024-02-19T20:40:38.487 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:38.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:38.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:38.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:38.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:38.257+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:38.488 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:38.258+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:38.502 INFO:teuthology.orchestra.run.smithi077.stdout:1708369200.0001512 mon.smithi077 (mon.0) 787 : cluster [WRN] Health detail: HEALTH_WARN 1 filesystem with deprecated feature inline_data 2024-02-19T20:40:38.502 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-19T20:40:38.503 INFO:tasks.cephadm:Compressing logs... 2024-02-19T20:40:38.503 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-19T20:40:38.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:40:38.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-19T20:40:38.569 INFO:teuthology.orchestra.run.smithi077.stderr:: No such file or directory 2024-02-19T20:40:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mgr.smithi077.yiiqpd.log 2024-02-19T20:40:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi077.ovassg.log 2024-02-19T20:40:38.570 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi077.jmjalp.log 2024-02-19T20:40:38.574 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mgr.smithi077.yiiqpd.log: /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi077.ovassg.log: gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.audit.log 2024-02-19T20:40:38.574 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi077.jmjalp.log: gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mon.smithi077.log 2024-02-19T20:40:38.574 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.1.log 2024-02-19T20:40:38.574 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mon.smithi077.log: 90.4% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.audit.log.gz 2024-02-19T20:40:38.574 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.0.log 2024-02-19T20:40:38.581 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.1.log: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-19T20:40:38.583 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.cephadm.log 2024-02-19T20:40:38.598 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.0.log: /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.cephadm.log: 83.9% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.cephadm.log.gz 2024-02-19T20:40:38.598 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.log 2024-02-19T20:40:38.608 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.log: 87.2%gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.2.log 2024-02-19T20:40:38.608 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph.log.gz 2024-02-19T20:40:38.620 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-volume.log 2024-02-19T20:40:38.660 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-volume.log: 91.1% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-volume.log.gz 2024-02-19T20:40:38.662 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi077.jmjalp.log: file size changed while zipping 2024-02-19T20:40:38.662 INFO:teuthology.orchestra.run.smithi077.stderr: 95.6% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi077.jmjalp.log.gz 2024-02-19T20:40:38.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:38.617+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:38.618146+0000) 2024-02-19T20:40:39.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:38.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:39.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:38.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:39.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:38.915+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:39.132 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:38.916+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:39.402 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:39.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:39.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:39.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:39.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:39.289+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:39.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:39.299+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.300302+0000) 2024-02-19T20:40:39.403 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:39.299+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:39.403 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:39.145+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.146050+0000) 2024-02-19T20:40:39.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:39.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:39.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:39.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:39.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:39.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:39.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:39.238+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:39.404 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:39.288+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.289515+0000) 2024-02-19T20:40:39.404 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:39.290+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69919 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:39.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:39.207+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.208042+0000) 2024-02-19T20:40:39.405 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:39.280+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.281965+0000) 2024-02-19T20:40:39.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:39.618+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:39.619778+0000) 2024-02-19T20:40:40.136 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:39.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:40.137 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:39.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:40.137 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:39.899+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:40.137 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:39.900+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:40.137 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:40.147+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:40.148192+0000) 2024-02-19T20:40:40.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:40.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:40.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:40.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:40.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:40.273+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:40.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:40.274+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:40.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:40.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:40.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:40.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:40.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:40.285+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:40.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:40.286+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:40.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:40.208+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:40.209705+0000) 2024-02-19T20:40:40.705 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mgr.smithi077.yiiqpd.log: file size changed while zipping 2024-02-19T20:40:40.706 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mgr.smithi077.yiiqpd.log.gz 2024-02-19T20:40:40.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:40.622+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:40.623756+0000) 2024-02-19T20:40:41.135 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:40.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:41.135 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:40.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:41.136 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:40.904+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:41.136 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:40.905+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:41.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:41.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:41.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:41.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:41.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:41.244+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:41.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:41.245+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:41.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:41.150+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.151323+0000) 2024-02-19T20:40:41.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:41.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:41.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:41.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:41.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:41.245+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:41.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:41.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:41.528 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:41.211+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.212439+0000) 2024-02-19T20:40:41.600 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:40:41.850 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:41.626+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:41.627946+0000) 2024-02-19T20:40:41.851 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:41.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:42.139 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:41.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:42.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:41.865+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:42.140 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:41.865+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:42.141 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:42.154+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:42.155229+0000) 2024-02-19T20:40:42.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:42.214+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:42.215195+0000) 2024-02-19T20:40:42.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:42.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:42.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:42.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:42.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:42.254+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:42.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:42.255+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:42.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:42.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:42.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:42.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:42.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:42.219+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:42.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:42.220+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:42.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:42.629+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:42.630503+0000) 2024-02-19T20:40:43.142 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:42.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:43.143 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:42.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:43.143 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:42.909+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:43.143 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:42.909+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:43.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:43.217+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:43.218299+0000) 2024-02-19T20:40:43.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:43.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:43.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:43.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:43.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:43.302+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:43.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:43.303+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:43.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:43.157+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:43.158332+0000) 2024-02-19T20:40:43.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:43.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:43.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:43.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:43.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:43.237+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:43.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:43.238+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:43.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:43.632+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:43.633661+0000) 2024-02-19T20:40:44.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:43.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:44.146 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:43.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:44.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:43.928+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:44.147 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:43 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:43.929+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:44.421 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:44.299+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69958 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:44.421 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:44.308+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.309923+0000) 2024-02-19T20:40:44.421 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:44.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:44.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:44.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:44.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:44.344+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:44.422 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:44.345+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:44.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:44.220+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.221304+0000) 2024-02-19T20:40:44.422 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:44.303+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.304832+0000) 2024-02-19T20:40:44.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:44.160+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.161374+0000) 2024-02-19T20:40:44.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:44.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:44.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:44.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:44.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:44.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:44.423 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:44.248+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:44.424 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:44.312+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.313862+0000) 2024-02-19T20:40:44.728 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-02-19T20:40:44.729 INFO:teuthology.orchestra.run.smithi077.stderr: 90.8% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.2.log.gz 2024-02-19T20:40:44.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:44.634+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:44.635202+0000) 2024-02-19T20:40:45.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:44.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:45.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:44.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:45.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:44.974+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:45.148 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:44 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:44.974+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:45.148 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:45.162+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:45.163622+0000) 2024-02-19T20:40:45.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:45.222+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:45.223043+0000) 2024-02-19T20:40:45.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:45.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:45.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:45.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:45.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:45.333+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:45.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:45.334+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:45.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:45.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:45.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:45.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:45.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:45.265+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:45.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:45.266+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:45.860 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:45.636+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:45.637458+0000) 2024-02-19T20:40:46.112 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-02-19T20:40:46.113 INFO:teuthology.orchestra.run.smithi077.stderr: 90.7% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.1.log.gz 2024-02-19T20:40:46.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:45.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:46.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:45.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:46.151 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:45.970+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:46.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:45 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:45.970+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:46.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:46.225+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.226094+0000) 2024-02-19T20:40:46.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:46.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:46.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:46.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:46.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:46.310+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:46.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:46.311+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:46.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:46.165+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.166769+0000) 2024-02-19T20:40:46.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:46.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:46.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:46.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:46.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:46.271+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:46.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:46.272+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:46.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:46.639+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:46.640415+0000) 2024-02-19T20:40:47.152 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-02-19T20:40:47.152 INFO:teuthology.orchestra.run.smithi077.stderr: 90.8% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-osd.0.log.gz 2024-02-19T20:40:47.154 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:47.169+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:47.170163+0000) 2024-02-19T20:40:47.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:46.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:47.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:46.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:47.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:46.984+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:47.155 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:46 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:46.984+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:47.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:47.227+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:47.228697+0000) 2024-02-19T20:40:47.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:47.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:47.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:47.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:47.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:47.297+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:47.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:47.298+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:47.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:47.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:47.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:47.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:47.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:47.331+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:47.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:47.332+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:47.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:47.642+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:47.643711+0000) 2024-02-19T20:40:48.158 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:47.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:48.158 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:47.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:48.158 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:47.963+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:48.158 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:47 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:47.963+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:48.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:48.231+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:48.232383+0000) 2024-02-19T20:40:48.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:48.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:48.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:48.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:48.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:48.290+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:48.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:48.290+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:48.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:48.172+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:48.173896+0000) 2024-02-19T20:40:48.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:48.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:48.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:48.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:48.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:48.274+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:48.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:48.275+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:48.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:48 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:48.646+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:48.647197+0000) 2024-02-19T20:40:49.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:49.008+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:49.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:49.008+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:49.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:49.008+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:49.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:49.009+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:49.415 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:49.309+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 69998 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:49.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:49.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:49.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:49.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:49.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:49.281+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:49.415 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:49.282+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:49.416 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:49.304+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.305315+0000) 2024-02-19T20:40:49.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:49.235+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.236043+0000) 2024-02-19T20:40:49.416 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:49.304+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.305988+0000) 2024-02-19T20:40:49.416 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:49.175+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.176447+0000) 2024-02-19T20:40:49.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:49.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:49.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:49.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:49.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:49.239+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:49.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:49.240+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:49.417 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:49.304+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.305684+0000) 2024-02-19T20:40:49.669 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:49 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:49.647+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:49.648712+0000) 2024-02-19T20:40:50.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:50.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:50.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:50.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:50.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:50.245+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:50.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:50.246+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:50.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:50.177+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:50.178704+0000) 2024-02-19T20:40:50.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:50.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:50.277 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:50.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:50.277 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:50.023+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:50.277 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:50.023+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:50.277 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:50.236+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:50.237539+0000) 2024-02-19T20:40:50.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:50.290+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:50.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:50.290+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:50.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:50.290+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:50.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:50.291+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:50.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:40:50.602 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:40:50.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:50 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:50.651+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:50.652028+0000) 2024-02-19T20:40:51.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:51.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:51.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:51.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:51.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:51.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:51.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:51.041+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:51.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:51.240+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:51.241040+0000) 2024-02-19T20:40:51.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:51.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:51.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:51.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:51.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:51.204+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:51.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:51.205+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:51.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:51.181+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:51.182253+0000) 2024-02-19T20:40:51.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:51.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:51.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:51.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:51.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:51.316+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:51.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:51.317+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:51.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:51 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:51.654+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:51.655338+0000) 2024-02-19T20:40:52.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:52.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:52.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:52.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:52.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:52.081+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:52.152 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:52.082+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:52.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:52.243+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:52.244501+0000) 2024-02-19T20:40:52.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:52.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:52.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:52.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:52.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:52.166+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:52.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:52.168+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:52.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:52.184+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:52.185791+0000) 2024-02-19T20:40:52.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:52.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:52.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:52.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:52.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:52.270+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:52.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:52.271+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:52.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:52 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:52.657+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:52.658740+0000) 2024-02-19T20:40:53.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:53.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:53.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:53.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:53.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:53.086+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:53.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:53.086+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:53.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:53.246+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:53.247927+0000) 2024-02-19T20:40:53.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:53.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:53.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:53.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:53.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:53.176+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:53.527 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:53.177+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:53.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:53.188+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:53.189465+0000) 2024-02-19T20:40:53.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:53.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:53.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:53.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:53.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:53.250+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:53.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:53.250+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:53.664 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:40:53.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:53 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:53.661+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:53.662235+0000) 2024-02-19T20:40:54.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:54.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:54.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:54.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:54.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:54.071+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:54.131 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:54.072+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:54.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:54.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:54.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:54.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:54.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:54.145+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:54.425 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:54.146+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:54.426 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:54.318+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:54.319065+0000) 2024-02-19T20:40:54.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:54.250+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:54.251484+0000) 2024-02-19T20:40:54.426 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:54.315+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:54.316183+0000) 2024-02-19T20:40:54.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:54.192+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:54.193042+0000) 2024-02-19T20:40:54.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:54.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:54.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:54.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:54.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:54.282+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:54.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:54.283+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:54.427 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:54.313+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:54.314727+0000) 2024-02-19T20:40:54.427 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:54.319+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 70037 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:54.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:54 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:54.663+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:54.664616+0000) 2024-02-19T20:40:55.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:55.067+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:55.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:55.067+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:55.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:55.067+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:55.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:55.067+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:55.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:55.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:55.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:55.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:55.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:55.110+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:55.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:55.111+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:55.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:55.252+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:55.253823+0000) 2024-02-19T20:40:55.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:55.194+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:55.195255+0000) 2024-02-19T20:40:55.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:55.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:55.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:55.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:55.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:55.241+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:55.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:55.242+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:55.861 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:55 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:55.667+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:55.668239+0000) 2024-02-19T20:40:56.005 INFO:teuthology.orchestra.run.smithi077.stderr:gzip: /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mon.smithi077.log: file size changed while zipping 2024-02-19T20:40:56.005 INFO:teuthology.orchestra.run.smithi077.stderr: 98.1% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mon.smithi077.log.gz 2024-02-19T20:40:56.160 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:56.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:56.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:56.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:56.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:56.075+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:56.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:56.076+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:56.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:56.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:56.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:56.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:56.162 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:56.034+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:56.162 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:56.035+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:56.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:56.254+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:56.255975+0000) 2024-02-19T20:40:56.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:56.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:56.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:56.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:56.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:56.192+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:56.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:56.193+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:56.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:56.196+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:56.197870+0000) 2024-02-19T20:40:56.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:56 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:56.670+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:56.671445+0000) 2024-02-19T20:40:57.157 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:57.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:57.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:57.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:57.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:57.073+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:57.158 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:57.073+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:57.158 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:57.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:57.159 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:57.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:57.159 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:57.040+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:57.159 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:57.041+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:57.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:57.258+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:57.259658+0000) 2024-02-19T20:40:57.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:57.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:57.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:57.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:57.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:57.172+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:57.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:57.173+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:57.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:57.200+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:57.201057+0000) 2024-02-19T20:40:57.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:57 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:57.672+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:57.673556+0000) 2024-02-19T20:40:58.160 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:58.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:58.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:58.083+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:58.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:58.084+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:58.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:58.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:58.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:58.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:58.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:58.103+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:58.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:58.103+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:58.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:58.260+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:58.261920+0000) 2024-02-19T20:40:58.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:58.202+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:58.203298+0000) 2024-02-19T20:40:58.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:58.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:58.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:58.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:58.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:58.203+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:58.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:58.204+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:58.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:58 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:58.675+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:58.676836+0000) 2024-02-19T20:40:59.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:59.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:59.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:59.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:59.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:59.119+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:40:59.161 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:59.119+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:59.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:59.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:59.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:59.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:59.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:59.057+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:40:59.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:59.058+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:40:59.431 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:40:59.328+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 70077 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:40:59.432 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:59.323+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:59.324540+0000) 2024-02-19T20:40:59.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:59.263+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:59.264626+0000) 2024-02-19T20:40:59.432 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:40:59.323+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:59.324177+0000) 2024-02-19T20:40:59.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:59.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:59.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:59.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:59.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:59.184+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:40:59.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:59.185+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:40:59.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:59.204+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:59.205432+0000) 2024-02-19T20:40:59.433 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:40:59.324+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:59.325502+0000) 2024-02-19T20:40:59.683 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:40:59 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:40:59.677+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:40:59.678334+0000) 2024-02-19T20:41:00.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:00.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:00.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:00.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:00.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:00.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:00.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:00.069+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:00.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:00.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:00.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:00.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:00.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:00.092+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:00.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:00.093+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:00.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:00.265+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:00.266101+0000) 2024-02-19T20:41:00.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:00.205+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:00.206785+0000) 2024-02-19T20:41:00.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:00.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:00.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:00.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:00.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:00.224+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:00.528 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:00.224+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:00.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:00 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:00.679+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:00.680463+0000) 2024-02-19T20:41:01.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:01.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:01.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:01.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:01.122 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:01.086+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:01.122 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:01.087+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:01.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:01.209+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:01.210104+0000) 2024-02-19T20:41:01.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:01.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:01.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:01.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:01.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:01.267+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:01.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:01.269+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:01.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:01.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:01.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:01.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:01.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:01.140+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:01.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:01.141+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:01.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:01.268+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:01.269431+0000) 2024-02-19T20:41:01.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:01 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:01.682+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:01.683874+0000) 2024-02-19T20:41:02.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:02.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:02.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:02.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:02.146 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:02.051+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:02.147 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:02.052+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:02.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:02.161+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:02.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:02.161+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:02.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:02.161+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:02.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:02.161+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:02.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:02.272+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:02.273189+0000) 2024-02-19T20:41:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:02.212+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:02.213575+0000) 2024-02-19T20:41:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:02.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:02.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:02.247+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:02.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:02.247+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:02.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:02 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:02.684+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:02.685893+0000) 2024-02-19T20:41:03.015 INFO:teuthology.orchestra.run.smithi077.stderr: 91.7% -- replaced with /var/log/ceph/6778a95a-cf57-11ee-95bb-87774f69a715/ceph-mds.cephfs.smithi077.ovassg.log.gz 2024-02-19T20:41:03.017 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-02-19T20:41:03.017 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m24.461s 2024-02-19T20:41:03.017 INFO:teuthology.orchestra.run.smithi077.stderr:user 1m4.598s 2024-02-19T20:41:03.017 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m1.522s 2024-02-19T20:41:03.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:03.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:03.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:03.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:03.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:03.089+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:03.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:03.090+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:03.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:03.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:03.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:03.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:03.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:03.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:03.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:03.201+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:03.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:03.274+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:03.275417+0000) 2024-02-19T20:41:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:03.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:03.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:03.209+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:03.210+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:03.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:03.215+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:03.216927+0000) 2024-02-19T20:41:03.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:03 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:03.688+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:03.689316+0000) 2024-02-19T20:41:04.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:04.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:04.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:04.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:04.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:04.059+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:04.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:04.060+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:04.439 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:41:04.336+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 70116 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:41:04.439 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:04.332+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:04.333429+0000) 2024-02-19T20:41:04.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:04.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:04.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:04.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:04.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:04.191+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:04.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:04.192+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:04.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:04.219+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:04.220411+0000) 2024-02-19T20:41:04.440 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:04.332+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:04.333931+0000) 2024-02-19T20:41:04.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:04.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:04.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:04.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:04.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:04.214+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:04.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:04.214+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:04.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:04.276+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:04.277708+0000) 2024-02-19T20:41:04.441 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:04.332+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:04.333589+0000) 2024-02-19T20:41:04.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:04 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:04.689+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:04.690781+0000) 2024-02-19T20:41:05.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:05.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:05.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:05.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:05.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:05.069+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:05.152 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:05.070+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:05.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:05.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:05.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:05.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:05.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:05.167+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:05.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:05.167+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:05.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:05.220+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:05.221819+0000) 2024-02-19T20:41:05.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:05.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:05.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:05.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:05.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:05.169+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:05.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:05.169+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:05.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:05.278+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:05.279221+0000) 2024-02-19T20:41:05.665 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:41:05.666 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:41:05.862 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:05 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:05.693+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:05.694219+0000) 2024-02-19T20:41:06.147 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:06.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:06.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:06.087+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:06.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:06.088+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:06.148 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:06.089+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:06.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:06.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:06.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:06.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:06.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:06.162+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:06.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:06.163+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:06.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:06.224+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:06.225287+0000) 2024-02-19T20:41:06.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:06.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:06.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:06.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:06.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:06.201+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:06.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:06.201+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:06.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:06.281+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:06.282630+0000) 2024-02-19T20:41:06.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:06 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:06.696+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:06.697565+0000) 2024-02-19T20:41:07.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:07.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:07.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:07.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:07.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:07.048+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:07.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:07.050+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:07.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:07.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:07.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:07.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:07.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:07.123+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:07.162 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:07.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:07.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:07.226+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:07.227274+0000) 2024-02-19T20:41:07.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:07.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:07.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:07.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:07.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:07.217+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:07.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:07.217+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:07.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:07.285+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:07.286096+0000) 2024-02-19T20:41:07.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:07 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:07.700+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:07.701070+0000) 2024-02-19T20:41:08.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:08.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:08.149 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:08.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:08.150 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:08.033+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:08.150 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:08.034+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:08.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:08.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:08.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:08.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:08.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:08.164+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:08.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:08.165+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:08.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:08.228+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:08.229261+0000) 2024-02-19T20:41:08.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:08.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:08.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:08.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:08.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:08.248+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:08.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:08.249+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:08.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:08.288+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:08.289626+0000) 2024-02-19T20:41:08.736 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:41:08.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:08 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:08.703+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:08.704537+0000) 2024-02-19T20:41:09.161 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:09.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:09.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:09.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:09.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:09.068+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:09.162 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:09.069+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:09.447 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:41:09.345+0000 7fe14334c700 -1 mon.smithi077@0(probing) e2 get_health_metrics reporting 70156 slow ops, oldest is monmgrreport(gid 14714, 0 checks, 1 progress events) 2024-02-19T20:41:09.448 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:09.340+0000 7f49eb38f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:09.341820+0000) 2024-02-19T20:41:09.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:09.291+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:09.292951+0000) 2024-02-19T20:41:09.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:09.292+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:09.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:09.292+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:09.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:09.292+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:09.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:09.292+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:09.449 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:09.341+0000 7f759db68700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:09.342784+0000) 2024-02-19T20:41:09.449 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:09.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:09.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:09.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:09.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:09.187+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:09.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:09.187+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:09.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:09.230+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:09.231287+0000) 2024-02-19T20:41:09.450 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:09.342+0000 7fb4d32bf700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:09.343784+0000) 2024-02-19T20:41:09.775 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:09 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:09.705+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:09.706857+0000) 2024-02-19T20:41:10.129 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:10.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:10.130 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:10.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:10.130 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:10.026+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:10.131 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:10.027+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:10.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:10.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:10.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:10.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:10.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:10.144+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:10.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:10.145+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:10.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:10.231+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:10.232781+0000) 2024-02-19T20:41:10.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:10.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:10.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:10.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:10.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:10.275+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:10.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:10.276+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:10.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:10.294+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:10.295210+0000) 2024-02-19T20:41:10.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:10 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:10.708+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:10.709319+0000) 2024-02-19T20:41:11.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:11.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:11.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:11.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:11.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:11.031+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:11.121 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:11.031+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:11.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:11.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:11.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:11.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:11.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:11.136+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:11.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:11.136+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:11.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:11.233+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:11.234905+0000) 2024-02-19T20:41:11.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:11.296+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:11.297239+0000) 2024-02-19T20:41:11.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:11.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:11.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:11.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:11.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:11.323+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:11.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:11.323+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:11.737 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:41:11.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:41:11.804 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:41:11.805 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:41:11.805 INFO:tasks.cephadm:Removing cluster... 2024-02-19T20:41:11.806 DEBUG:teuthology.orchestra.run.smithi077:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 --force 2024-02-19T20:41:11.863 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:11 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:11.710+0000 7f49de7af700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:11.711965+0000) 2024-02-19T20:41:12.162 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:12 smithi077 systemd[1]: Stopping Ceph mon.smithi077 for 6778a95a-cf57-11ee-95bb-87774f69a715... 2024-02-19T20:41:12.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:12.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:12.163 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:12.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:12.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:12.020+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:12.164 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:12.021+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:12.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:12.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:12.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:12.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:12.164 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:12.122+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:12.165 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:12.123+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:12.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:12.235+0000 7fb4c6ee0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:12.236878+0000) 2024-02-19T20:41:12.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:41:12.410+0000 7fe146352700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi077 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:41:12.526 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077[29243]: 2024-02-19T20:41:12.410+0000 7fe146352700 -1 mon.smithi077@0(probing) e2 *** Got Signal Terminated *** 2024-02-19T20:41:12.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:12.299+0000 7f758f785700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-02-19T19:41:12.299361+0000) 2024-02-19T20:41:12.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:12.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:12.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:12.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:12.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:12.358+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:12.527 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:12 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:12.358+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:13.324 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:13 smithi077 podman[121915]: 2024-02-19 20:41:13.256205933 +0000 UTC m=+1.002232162 container died 7b41d21d0c00d3c4f4079ac54af420c422027c36ed01eb8819ac061178f18628 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, ceph=True, RELEASE=quincy-c80e716, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T20:41:13.325 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:13.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:13.325 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:13.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:13.325 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:13.044+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:13.325 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:13.045+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:13.325 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:13.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:13.325 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:13.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:13.325 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:13.140+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:13.326 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:13.140+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:13.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:13.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:13.680 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:13.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:13.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:13.340+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:13.681 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:13 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:13.340+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:14.025 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:13 smithi077 podman[121915]: 2024-02-19 20:41:13.696491341 +0000 UTC m=+1.442517615 container cleanup 7b41d21d0c00d3c4f4079ac54af420c422027c36ed01eb8819ac061178f18628 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T20:41:14.026 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:13 smithi077 bash[121915]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077 2024-02-19T20:41:14.026 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:13 smithi077 podman[121929]: 2024-02-19 20:41:13.856182448 +0000 UTC m=+0.601968923 container remove 7b41d21d0c00d3c4f4079ac54af420c422027c36ed01eb8819ac061178f18628 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-mon-smithi077, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=quincy-c80e716, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-19T20:41:14.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:14 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi077.service: Deactivated successfully. 2024-02-19T20:41:14.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:14 smithi077 systemd[1]: Stopped Ceph mon.smithi077 for 6778a95a-cf57-11ee-95bb-87774f69a715. 2024-02-19T20:41:14.333 INFO:journalctl@ceph.mon.smithi077.smithi077.stdout:Feb 19 20:41:14 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@mon.smithi077.service: Consumed 1h 14min 56.298s CPU time. 2024-02-19T20:41:14.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:14.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:14.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:14.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:14.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:14.104+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:14.334 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:14.104+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:14.334 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:14.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:14.335 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:14.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:14.335 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:14.061+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:14.335 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:14.062+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:14.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:14.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:14.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:14.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:14.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:14.348+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:14.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:14 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:14.348+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:15.170 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:15.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:15.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:15.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:15.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:15.024+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:15.171 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:15.025+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:15.172 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:15.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:15.172 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:15.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:15.172 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:15.112+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:15.173 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:15.113+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:15.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:15.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:15.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:15.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:15.505 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:15.319+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:15.506 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:15 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:15.320+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:16.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:16.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:16.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:16.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:16.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:16.036+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:16.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:16.037+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:16.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:16.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:16.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:16.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:16.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:16.075+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:16.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:16.075+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:16.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:16.351+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:16.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:16.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:16.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:16.352+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:16.746 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:16 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:16.352+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:17.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:17.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:17.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:17.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:17.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:17.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:17.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:17.001+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:17.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:17.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:17.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:17.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:17.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:17.100+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:17.377 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:17.101+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:17.707 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:17.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:17.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:17.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:17.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:17.392+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:17.708 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:17.392+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:18.025 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:17.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:18.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:17.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:18.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:17.977+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:18.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:17 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:17.977+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:18.425 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:18.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:18.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:18.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:18.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:18.083+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:18.426 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:18.083+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:18.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:18.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:18.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:18.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:18.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:18.440+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:18.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:18 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:18.441+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:19.257 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:19.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:19.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:19.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:19.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:19.034+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:19.258 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:19.035+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:19.258 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:19.025+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:19.258 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:19.025+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:19.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:19.025+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:19.259 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:19.025+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:19.523 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:19.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:19.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:19.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:19.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:19.485+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:19.524 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:19 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:19.486+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:20.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:20.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:20.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:20.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:20.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:20.039+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:20.125 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:20.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:20.125 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:20.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:20.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:20.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:20.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:20.000+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:20.126 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:20.001+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:20.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:20.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:20.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:20.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:20.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:20.535+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:20.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:20.536+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:21.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:20.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:21.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:20.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:21.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:20.952+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:21.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:20 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:20.953+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:21.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:21.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:21.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:21.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:21.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:21.038+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:21.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:21.039+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:21.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:21.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:21.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:21.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:21.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:21.522+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:21.609 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:21.522+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:21.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:21 smithi077 systemd[1]: Stopping Ceph osd.0 for 6778a95a-cf57-11ee-95bb-87774f69a715... 2024-02-19T20:41:21.969 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:21.862+0000 7f49e938b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:41:21.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:21.862+0000 7f49e938b700 -1 osd.0 48 *** Got signal Terminated *** 2024-02-19T20:41:21.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:21.862+0000 7f49e938b700 -1 osd.0 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:41:21.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:21.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:21.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:21.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:21.970 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:21.984+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:22.275 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:21 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:21.985+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:22.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:22.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:22.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:22.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:22.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:22.006+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:22.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:22.006+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:22.946 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:22.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:22.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:22.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:22.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:22.553+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:22.947 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:22.553+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:23.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:23.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:23.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:23.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:23.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:23.056+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:23.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:23.057+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:23.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:22.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:23.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:22.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:23.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:22.961+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:23.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:22 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:22.962+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:23.986 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:23.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:23.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:23.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:23.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:23.557+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:23.987 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:23 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:23.557+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:24.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:24.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:24.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:24.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:24.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:24.067+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:24.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:24.067+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:24.276 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:24.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:24.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:24.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:24.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:24.001+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:24.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:24.002+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:24.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:24.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:24.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:24.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:24.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:24.573+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:24.814 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:24.573+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:25.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:24.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:25.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:24.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:25.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:24.976+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:25.098 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:24 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:24.977+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:25.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:25.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:25.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:25.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:25.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:25.113+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:25.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:25.114+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:25.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:25.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:25.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:25.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:25.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:25.590+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:25.950 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:25.590+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:26.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:26.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:26.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:26.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:26.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:26.119+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:26.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:26.120+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:26.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:25.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:26.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:25.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:26.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:25.965+0000 7f49e718c700 -1 osd.0 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:38:49.065466+0000 (oldest deadline 2024-02-19T20:39:09.065466+0000) 2024-02-19T20:41:26.277 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:25 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0[47889]: 2024-02-19T20:41:25.966+0000 7f49e718c700 -1 osd.0 48 get_health_metrics reporting 249 slow ops, oldest is osd_op(client.14578.0:9172 3.17 3:eea1a6fa:::1000000507e.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:27.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:26.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:27.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:26.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:27.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:26.637+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:27.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:26 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:26.638+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:27.526 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:27 smithi077 podman[122768]: 2024-02-19 20:41:27.330419885 +0000 UTC m=+5.569837408 container died 7a663ccf6c07f0621c8c75baef2ff8b8a6cb49a8b6c8b3e8d93aa4694b879217 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0, RELEASE=quincy-c80e716, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-19T20:41:27.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:27.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:27.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:27.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:27.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:27.134+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:27.527 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:27.135+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:28.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:27 smithi077 podman[122768]: 2024-02-19 20:41:27.733269656 +0000 UTC m=+5.972687189 container cleanup 7a663ccf6c07f0621c8c75baef2ff8b8a6cb49a8b6c8b3e8d93aa4694b879217 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-c80e716) 2024-02-19T20:41:28.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:27 smithi077 bash[122768]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0 2024-02-19T20:41:28.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:27.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:28.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:27.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:28.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:27.671+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:28.027 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:27 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:27.671+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:28.467 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:28 smithi077 podman[122787]: 2024-02-19 20:41:28.081591447 +0000 UTC m=+0.746253760 container remove 7a663ccf6c07f0621c8c75baef2ff8b8a6cb49a8b6c8b3e8d93aa4694b879217 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, GIT_CLEAN=True, RELEASE=quincy-c80e716, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0) 2024-02-19T20:41:28.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:28.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:28.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:28.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:28.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:28.090+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:28.467 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:28.091+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:28.744 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:28.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:28.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:28.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:28.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:28.685+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:28.745 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:28 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:28.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:28.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:28 smithi077 podman[122878]: 2024-02-19 20:41:28.482862368 +0000 UTC m=+0.099143860 container create 9318cd9cb3489f2c19a37ef9c2f81009b4a7e94cc0e8ffa9143ac6af80b4ef1c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=quincy-c80e716, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-02-19T20:41:28.745 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:28 smithi077 podman[122878]: 2024-02-19 20:41:28.402201383 +0000 UTC m=+0.018482849 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-19T20:41:28.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:28 smithi077 podman[122878]: 2024-02-19 20:41:28.668531195 +0000 UTC m=+0.284812641 container init 9318cd9cb3489f2c19a37ef9c2f81009b4a7e94cc0e8ffa9143ac6af80b4ef1c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-19T20:41:28.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:28 smithi077 podman[122878]: 2024-02-19 20:41:28.671637482 +0000 UTC m=+0.287918929 container start 9318cd9cb3489f2c19a37ef9c2f81009b4a7e94cc0e8ffa9143ac6af80b4ef1c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=quincy-c80e716, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-19T20:41:28.746 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:28 smithi077 podman[122878]: 2024-02-19 20:41:28.76022004 +0000 UTC m=+0.376501504 container attach 9318cd9cb3489f2c19a37ef9c2f81009b4a7e94cc0e8ffa9143ac6af80b4ef1c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, RELEASE=quincy-c80e716, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-19T20:41:29.026 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:28 smithi077 podman[122878]: 2024-02-19 20:41:28.967824069 +0000 UTC m=+0.584105526 container died 9318cd9cb3489f2c19a37ef9c2f81009b4a7e94cc0e8ffa9143ac6af80b4ef1c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214) 2024-02-19T20:41:29.514 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:29.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:29.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:29.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:29.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:29.042+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:29.515 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:29.042+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:29.775 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:29 smithi077 systemd[1]: Stopping Ceph osd.1 for 6778a95a-cf57-11ee-95bb-87774f69a715... 2024-02-19T20:41:29.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:29 smithi077 podman[122906]: 2024-02-19 20:41:29.530368693 +0000 UTC m=+0.554232641 container remove 9318cd9cb3489f2c19a37ef9c2f81009b4a7e94cc0e8ffa9143ac6af80b4ef1c (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=) 2024-02-19T20:41:29.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:29 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.0.service: Deactivated successfully. 2024-02-19T20:41:29.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:29 smithi077 systemd[1]: Stopped Ceph osd.0 for 6778a95a-cf57-11ee-95bb-87774f69a715. 2024-02-19T20:41:29.776 INFO:journalctl@ceph.osd.0.smithi077.stdout:Feb 19 20:41:29 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.0.service: Consumed 56.011s CPU time. 2024-02-19T20:41:29.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:29.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:29.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:29.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:29.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:29.686+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:29.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:29.686+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:30.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:29.991+0000 7fb4d12bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:41:30.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:29.991+0000 7fb4d12bb700 -1 osd.1 48 *** Got signal Terminated *** 2024-02-19T20:41:30.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:29 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:29.991+0000 7fb4d12bb700 -1 osd.1 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:41:30.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:30.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:30.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:30.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:30.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:30.023+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:30.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:30.024+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:31.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:30.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:31.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:30.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:31.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:30.706+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:31.023 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:30 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:30.706+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:31.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:31.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:31.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:31.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:31.276 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:31.037+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:31.277 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:31.038+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:32.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:31.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:32.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:31.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:32.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:31.734+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:32.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:31 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:31.734+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:32.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:32.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:32.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:32.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:32.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:32.080+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:32.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:32.081+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:33.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:32.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:33.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:32.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:33.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:32.722+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:33.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:32 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:32.722+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:33.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:33.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:33.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:33.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:33.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:33.062+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:33.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:33.063+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:34.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:33.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:34.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:33.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:34.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:33.688+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:34.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:33 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:33.688+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:34.525 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:34.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:34.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:34.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:34.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:34.050+0000 7fb4ce8bb700 -1 osd.1 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:16.238100+0000 (oldest deadline 2024-02-19T20:39:36.238100+0000) 2024-02-19T20:41:34.526 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1[55295]: 2024-02-19T20:41:34.051+0000 7fb4ce8bb700 -1 osd.1 48 get_health_metrics reporting 237 slow ops, oldest is osd_op(mds.0.4:836 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=193169b,omap-rm-keys in=1354b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e48) 2024-02-19T20:41:35.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:34.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:35.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:34.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:35.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:34.639+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:35.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:34 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:34.640+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:35.860 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:35 smithi077 podman[122995]: 2024-02-19 20:41:35.598405362 +0000 UTC m=+5.709672646 container died c0fec34c6b1c774cca3724ce609a8ef06ebf063c2338ba914d2488c18ef2e871 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=quincy-c80e716, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, org.label-schema.schema-version=1.0) 2024-02-19T20:41:35.861 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:35 smithi077 podman[122995]: 2024-02-19 20:41:35.876189932 +0000 UTC m=+5.987457268 container cleanup c0fec34c6b1c774cca3724ce609a8ef06ebf063c2338ba914d2488c18ef2e871 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240214, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-02-19T20:41:35.861 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:35.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:35.862 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:35.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:35.862 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:35.630+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:35.862 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:35 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:35.630+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:36.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:35 smithi077 bash[122995]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1 2024-02-19T20:41:36.275 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:36 smithi077 podman[123015]: 2024-02-19 20:41:36.102512798 +0000 UTC m=+0.500290360 container remove c0fec34c6b1c774cca3724ce609a8ef06ebf063c2338ba914d2488c18ef2e871 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=quincy-c80e716, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.build-date=20240214) 2024-02-19T20:41:36.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:36.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:36.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:36.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:36.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:36.600+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:36.673 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:36 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:36.600+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:36.673 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:36 smithi077 podman[123108]: 2024-02-19 20:41:36.322570771 +0000 UTC m=+0.018323054 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-19T20:41:36.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:36 smithi077 podman[123108]: 2024-02-19 20:41:36.436805352 +0000 UTC m=+0.132557642 container create 9ca3b40d8df60af57cf4531741d1f5188d29e3f842eb140fb97d6977253d6bc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-19T20:41:36.674 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:36 smithi077 podman[123108]: 2024-02-19 20:41:36.689130543 +0000 UTC m=+0.384882827 container init 9ca3b40d8df60af57cf4531741d1f5188d29e3f842eb140fb97d6977253d6bc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:41:36.951 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:36 smithi077 podman[123108]: 2024-02-19 20:41:36.69218067 +0000 UTC m=+0.387932954 container start 9ca3b40d8df60af57cf4531741d1f5188d29e3f842eb140fb97d6977253d6bc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, RELEASE=quincy-c80e716, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214) 2024-02-19T20:41:36.952 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:36 smithi077 podman[123108]: 2024-02-19 20:41:36.780777258 +0000 UTC m=+0.476529563 container attach 9ca3b40d8df60af57cf4531741d1f5188d29e3f842eb140fb97d6977253d6bc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240214, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-19T20:41:37.234 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:36 smithi077 podman[123108]: 2024-02-19 20:41:36.96778342 +0000 UTC m=+0.663535708 container died 9ca3b40d8df60af57cf4531741d1f5188d29e3f842eb140fb97d6977253d6bc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=quincy-c80e716, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240214) 2024-02-19T20:41:37.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:37 smithi077 podman[123134]: 2024-02-19 20:41:37.467663079 +0000 UTC m=+0.494922909 container remove 9ca3b40d8df60af57cf4531741d1f5188d29e3f842eb140fb97d6977253d6bc3 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=quincy-c80e716, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T20:41:37.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:37 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.1.service: Deactivated successfully. 2024-02-19T20:41:37.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:37 smithi077 systemd[1]: Stopped Ceph osd.1 for 6778a95a-cf57-11ee-95bb-87774f69a715. 2024-02-19T20:41:37.492 INFO:journalctl@ceph.osd.1.smithi077.stdout:Feb 19 20:41:37 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.1.service: Consumed 56.813s CPU time. 2024-02-19T20:41:37.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:37.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:37.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:37.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:37.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:37.626+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:37.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:37.626+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:37.777 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:37 smithi077 systemd[1]: Stopping Ceph osd.2 for 6778a95a-cf57-11ee-95bb-87774f69a715... 2024-02-19T20:41:38.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:37.911+0000 7f759bb64700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-19T20:41:38.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:37.911+0000 7f759bb64700 -1 osd.2 48 *** Got signal Terminated *** 2024-02-19T20:41:38.276 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:37 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:37.911+0000 7f759bb64700 -1 osd.2 48 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-19T20:41:39.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:38.615+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:39.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:38.615+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:39.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:38.615+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:39.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:38 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:38.615+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:39.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:39.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:39.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:39.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:39.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:39.606+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:39.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:39 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:39.607+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:40.775 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:40.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:40.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:40.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:40.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:40.588+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:40.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:40 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:40.588+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:42.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:41.636+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:42.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:41.636+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:42.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:41.636+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:42.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:41 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:41.636+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:43.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:42.621+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6804 osd.3 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:43.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:42.621+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6812 osd.4 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:43.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:42.621+0000 7f7597160700 -1 osd.2 48 heartbeat_check: no reply from 172.21.15.104:6820 osd.5 ever on either front or back, first ping sent 2024-02-19T20:39:03.756802+0000 (oldest deadline 2024-02-19T20:39:23.756802+0000) 2024-02-19T20:41:43.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:42 smithi077 ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2[63037]: 2024-02-19T20:41:42.621+0000 7f7597160700 -1 osd.2 48 get_health_metrics reporting 145 slow ops, oldest is osd_op(client.14578.0:9168 3.1e 3:78dc8545:::10000005281.00000000:head [write 0~915 [1@-1] in=915b] snapc 1=[] ondisk+write+known_if_redirected e48) 2024-02-19T20:41:43.596 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:43 smithi077 podman[123213]: 2024-02-19 20:41:43.308577369 +0000 UTC m=+5.496444690 container died 0f448003509d7319c10997e7f9cf643cd66ff3f2f32aa2e14dcd5bd2edf1b2d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-19T20:41:44.025 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:43 smithi077 podman[123213]: 2024-02-19 20:41:43.612657647 +0000 UTC m=+5.800524975 container cleanup 0f448003509d7319c10997e7f9cf643cd66ff3f2f32aa2e14dcd5bd2edf1b2d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD, org.label-schema.build-date=20240214, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-19T20:41:44.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:43 smithi077 bash[123213]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2 2024-02-19T20:41:44.026 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:43 smithi077 podman[123242]: 2024-02-19 20:41:43.82221459 +0000 UTC m=+0.509422616 container remove 0f448003509d7319c10997e7f9cf643cd66ff3f2f32aa2e14dcd5bd2edf1b2d5 (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd) 2024-02-19T20:41:44.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:44 smithi077 podman[123335]: 2024-02-19 20:41:44.122354023 +0000 UTC m=+0.017538844 image pull 10b0c032c913004d1f4f6c0f2496f5bfddfde4aeddb489f4129f49e49277ea3e quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b 2024-02-19T20:41:44.485 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:44 smithi077 podman[123335]: 2024-02-19 20:41:44.231772749 +0000 UTC m=+0.126957554 container create 78b462964ed8e6a393c6b3dfb1c90136360894382f0dc11b1f29bdfa140ea6bc (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240214, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=quincy-c80e716, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-02-19T20:41:44.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:44 smithi077 podman[123335]: 2024-02-19 20:41:44.43422003 +0000 UTC m=+0.329404836 container init 78b462964ed8e6a393c6b3dfb1c90136360894382f0dc11b1f29bdfa140ea6bc (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=quincy-c80e716, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240214, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-02-19T20:41:44.486 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:44 smithi077 podman[123335]: 2024-02-19 20:41:44.437316334 +0000 UTC m=+0.332501143 container start 78b462964ed8e6a393c6b3dfb1c90136360894382f0dc11b1f29bdfa140ea6bc (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240214, RELEASE=quincy-c80e716, GIT_CLEAN=True) 2024-02-19T20:41:44.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:44 smithi077 podman[123335]: 2024-02-19 20:41:44.500747934 +0000 UTC m=+0.395932748 container attach 78b462964ed8e6a393c6b3dfb1c90136360894382f0dc11b1f29bdfa140ea6bc (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240214, RELEASE=quincy-c80e716, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, maintainer=Guillaume Abrioux ) 2024-02-19T20:41:44.776 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:44 smithi077 podman[123335]: 2024-02-19 20:41:44.720987004 +0000 UTC m=+0.616171813 container died 78b462964ed8e6a393c6b3dfb1c90136360894382f0dc11b1f29bdfa140ea6bc (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240214, GIT_CLEAN=True) 2024-02-19T20:41:45.525 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:45 smithi077 podman[123361]: 2024-02-19 20:41:45.254239894 +0000 UTC m=+0.524179193 container remove 78b462964ed8e6a393c6b3dfb1c90136360894382f0dc11b1f29bdfa140ea6bc (image=quay.ceph.io/ceph-ci/ceph@sha256:479114d4af9dd6c9b2771de62c79e08a4e4ba3cecb535f7d44dca6fc106bb04b, name=ceph-6778a95a-cf57-11ee-95bb-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=quincy-c80e716, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240214, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=1617517b9622744995c4661989e3c30d036a7cfd, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-02-19T20:41:45.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:45 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.2.service: Deactivated successfully. 2024-02-19T20:41:45.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:45 smithi077 systemd[1]: Stopped Ceph osd.2 for 6778a95a-cf57-11ee-95bb-87774f69a715. 2024-02-19T20:41:45.526 INFO:journalctl@ceph.osd.2.smithi077.stdout:Feb 19 20:41:45 smithi077 systemd[1]: ceph-6778a95a-cf57-11ee-95bb-87774f69a715@osd.2.service: Consumed 51.616s CPU time. 2024-02-19T20:41:50.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:41:50.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:41:53.757 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:42:02.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:42:02.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:05.820 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:42:17.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:42:17.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:20.892 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:42:23.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:42:23.894 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:23.964 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:42:23.964 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:42:23.965 INFO:tasks.cephadm:Teardown complete 2024-02-19T20:42:23.965 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:42:23.991 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-19T20:42:24.006 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-02-19T20:42:24.024 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-19T20:42:24.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-19T20:42:24.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:42:24.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:27.104 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:42:36.105 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:42:36.106 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:39.168 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:42:51.169 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:42:51.170 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:54.240 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:42:57.241 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:42:57.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:57.308 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:42:57.308 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:42:57.349 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:42:57.377 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-19T20:42:57.389 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-19T20:42:57.389 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-19T20:42:57.397 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:42:57.398 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:42:57.412 INFO:teuthology.orchestra.run.smithi077.stderr:bash: line 1: ntpq: command not found 2024-02-19T20:42:57.417 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-19T20:42:57.417 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2024-02-19T20:42:57.417 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 +410us[ +429us] +/- 88ms 2024-02-19T20:42:57.417 INFO:teuthology.orchestra.run.smithi077.stdout:^* hv02.front.sepia.ceph.com 3 7 377 4 -769us[ -751us] +/- 71ms 2024-02-19T20:42:57.418 INFO:teuthology.orchestra.run.smithi077.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 6 +795us[ +814us] +/- 95ms 2024-02-19T20:42:57.418 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-02-19T20:43:00.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:43:09.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:43:09.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:12.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:43:24.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:43:24.543 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:27.616 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:43:30.617 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:43:30.618 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:30.684 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:43:30.684 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:43:30.685 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:43:30.709 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-19T20:43:30.726 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-19T20:43:30.726 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-19T20:43:30.739 DEBUG:teuthology.orchestra.run.smithi077:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-19T20:43:30.832 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:43:30.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:33.884 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:43:42.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:43:42.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:43:45.952 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:43:57.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:43:57.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:01.024 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:44:04.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:44:04.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:04.092 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:44:04.092 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:44:04.093 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:44:04.101 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-19T20:44:04.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-19T20:44:04.123 INFO:teuthology.task.internal:Duration was 7452.003729 seconds 2024-02-19T20:44:04.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-19T20:44:04.135 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-19T20:44:04.135 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-19T20:44:04.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:44:04.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:04.191 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-19T20:44:07.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:44:16.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:44:16.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:19.264 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:44:31.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:44:31.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:34.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:44:37.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:44:37.333 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:37.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:44:37.404 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:44:37.405 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:44:37.420 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-19T20:44:37.432 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-19T20:44:37.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-19T20:44:37.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:44:37.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:40.540 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:44:49.541 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:44:49.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:44:52.604 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:45:04.606 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:45:04.607 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:07.680 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:45:10.681 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:45:10.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:10.748 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:45:10.748 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:45:10.749 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:45:10.764 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-19T20:45:10.779 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-19T20:45:10.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:45:10.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:10.814 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-02-19T20:45:13.852 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:45:22.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:45:22.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:25.916 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:45:37.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:45:37.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:40.988 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:45:43.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:45:43.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:44.060 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:45:44.060 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:45:44.061 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:45:44.079 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-19T20:45:44.091 INFO:teuthology.task.internal:Transferring archived files... 2024-02-19T20:45:44.092 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617/remote/smithi077 2024-02-19T20:45:44.093 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-19T20:45:44.139 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617/remote/smithi104 2024-02-19T20:45:44.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:45:44.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:47.196 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:45:56.197 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:45:56.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:45:59.260 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:46:11.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:46:11.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:14.332 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:46:17.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:46:17.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:17.404 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:46:17.404 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:46:17.404 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:46:17.421 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-19T20:46:17.434 INFO:teuthology.task.internal:Not uploading archives. 2024-02-19T20:46:17.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-19T20:46:17.446 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-19T20:46:17.446 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-19T20:46:17.453 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:46:17.453 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:17.468 INFO:teuthology.orchestra.run.smithi077.stdout: 265295 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 19 20:42 /home/ubuntu/cephtest 2024-02-19T20:46:17.469 INFO:teuthology.orchestra.run.smithi077.stdout: 265296 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Feb 19 20:45 /home/ubuntu/cephtest/archive 2024-02-19T20:46:17.469 INFO:teuthology.orchestra.run.smithi077.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 18:39 /home/ubuntu/cephtest/archive/syslog 2024-02-19T20:46:17.469 INFO:teuthology.orchestra.run.smithi077.stdout: 265302 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 19 18:39 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-19T20:46:17.469 INFO:teuthology.orchestra.run.smithi077.stdout: 265301 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Feb 19 18:39 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-19T20:46:17.469 INFO:teuthology.orchestra.run.smithi077.stdout: 402313 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 19 20:43 /home/ubuntu/cephtest/archive/audit 2024-02-19T20:46:17.469 INFO:teuthology.orchestra.run.smithi077.stdout: 426286 200 -rw------- 1 ubuntu ubuntu 203747 Feb 19 20:43 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-02-19T20:46:17.469 INFO:teuthology.orchestra.run.smithi077.stdout: 265380 360 -rwxr-xr-x 1 ubuntu ubuntu 367151 Feb 19 18:48 /home/ubuntu/cephtest/cephadm 2024-02-19T20:46:17.470 INFO:teuthology.orchestra.run.smithi077.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-02-19T20:46:20.508 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:46:29.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:46:29.510 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:32.572 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:46:44.573 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:46:44.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:47.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:46:50.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-02-19T20:46:50.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-19T20:46:50.720 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-02-19T20:46:50.720 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-02-19T20:46:50.721 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/github.com_batrick_ceph_1ce7ea8877c709fa862eac608550374bd96e166e/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_b1dac5519c57c269ea98a22fb7729016a1d0e4d2/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi104 2024-02-19T20:46:50.737 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-19T20:46:50.751 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-19T20:46:50.764 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-02-19T20:46:50.764 INFO:teuthology.nuke:Checking targets against current locks 2024-02-19T20:46:50.796 DEBUG:teuthology.nuke:shortname: smithi077 2024-02-19T20:46:50.797 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T20:46:50.822 DEBUG:teuthology.nuke:shortname: smithi104 2024-02-19T20:46:50.822 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-19T20:46:50.825 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:48.285546', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM2yDy7yT4TWsPg0hMNtmQ5bTfrGOnlQy57t5RFdWNXEzxamh4TpvjzKRgdv44arNbjXve1OkYOk5Uh5vwhRJ+8='} 2024-02-19T20:46:50.854 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566617', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-19 18:29:48.286859', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsTJiI/5NZz48HT59dZd4kjS1jf7gbBr7c25EYpQKt7xHJOrVyhw+MGQXHSQsGLi4ISYOBIKKLOnbJv4yvOxyc='} 2024-02-19T20:46:50.881 INFO:teuthology.orchestra.console.smithi077:Power off 2024-02-19T20:46:50.881 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T20:46:50.904 DEBUG:teuthology.orchestra.console.smithi077:power off output: Chassis Power Control: Down/Off 2024-02-19T20:46:50.905 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:50.923 INFO:teuthology.orchestra.console.smithi104:Power off 2024-02-19T20:46:50.924 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-19T20:46:50.932 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-02-19T20:46:50.945 DEBUG:teuthology.orchestra.console.smithi104:power off output: Chassis Power Control: Down/Off 2024-02-19T20:46:50.945 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:50.968 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-02-19T20:46:54.934 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:54.969 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:55.046 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is on 2024-02-19T20:46:55.079 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is on 2024-02-19T20:46:59.052 DEBUG:teuthology.orchestra.console.smithi077:pexpect command: ipmitool -H smithi077.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:59.080 DEBUG:teuthology.orchestra.console.smithi104:pexpect command: ipmitool -H smithi104.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-19T20:46:59.162 DEBUG:teuthology.orchestra.console.smithi077:check power output: Chassis Power is off 2024-02-19T20:46:59.190 DEBUG:teuthology.orchestra.console.smithi104:check power output: Chassis Power is off 2024-02-19T20:46:59.263 INFO:teuthology.orchestra.console.smithi077:Power off completed 2024-02-19T20:46:59.291 INFO:teuthology.orchestra.console.smithi104:Power off completed 2024-02-19T20:46:59.397 INFO:teuthology.run:Summary data: description: fs:upgrade:mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 0a 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 7452.003728866577 failure_reason: 'Command failed on smithi077 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 6778a95a-cf57-11ee-95bb-87774f69a715 -e sha1=f78a58c0ffd401d1493058a1022c35f011d65275 -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fd340e7905d94c4bbfe20140ba0ea85f status: fail success: false 2024-02-19T20:46:59.398 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-19T20:46:59.477 INFO:teuthology.run:FAIL